Building Multilingual Machine Translation Systems
That Serve Arbitrary X-Y Translations

Multilingual Neural Machine Translation (MNMT), which enables one system to serve translation for multiple directions, has attracted much attention in the machine translation area Zoph and Knight (2016); Firat et al. (2016). Because the multilingual capability hugely reduces the deployment cost at training and inference, MNMT has actively been employed as a machine translation system backbone in recent years Johnson et al. (2017); Hassan et al. (2018).

Most MNMT systems are trained with multiple English-centric data for both directions (e.g., English $\rightarrow$ {French, Chinese} (En-X) and {French, Chinese} $\rightarrow$ English (X-En)). Recent work Gu et al. (2019); Zhang et al. (2020); Yang et al. (2021b) pointed out that such MNMT systems severely face an off-target translation issue, especially in translations from a non-English language X to another non-English language Y. Meanwhile, Freitag and Firat (2020) have extended data resources with multi-way aligned data and reported that one complete many-to-many MNMT can be fully supervised, achieving competitive translation performance for all X-Y directions. In our preliminary experiments, we observed that the complete many-to-many training is still as challenging as one-to-many training Johnson et al. (2017); Wang et al. (2020), since we have introduced more one-to-many translation tasks into the training. Similarly reported in the many-to-many training with zero-shot setup Gu et al. (2019); Yang et al. (2021b), the complete MNMT model also suffers from capturing correlations in the data for all the X-Y directions as one model training, due to highly imbalanced data.

In this paper, we propose a two-stage training for complete MNMT systems that serve arbitrary X-Y translations by 1) pretraining a complete multilingual many-to-many model and 2) finetuning the model to effectively transfer knowledge from pretraining to task-specific multilingual systems. Considering that MNMT is a multi-task learner of translation tasks with “multiple languages”, the complete multilingual model learns more diverse and general multilingual representations. We transfer the representations to a specifically targeted task via many-to-one multilingual finetuning, and eventually build multiple many-to-one MNMT models that cover all X-Y directions. The experimental results on the WMT’21 multilingual translation task show that our systems have substantial improvement against conventional bilingual approaches and many-to-one multilingual approaches for most directions. Besides, we discuss our proposal in the light of feasible deployment scenarios and show that the proposed approach also works well in an extremely large-scale data setting.

To support all possible translations with $|L|$ languages (including English), we first train a complete MNMT system on all available parallel data for $|L|\times(|L|-1)$ directions. We assume that there exist data of $(|L|-1)$ English-centric language pairs and remaining $\frac{(|L|-1)\times(|L|-2)}{2}$ non-English-centric language pairs, which lets the system learn multilingual representations across all $|L|$ languages. Usually, the volume of English-centric data is much greater than non-English-centric one. Then, we transfer the multilingual representations to one target language $L$ by finetuning the system on a subset of training data for many-to-$L$ directions (i.e., multilingual many-to-one finetuning). This step leads the decoder towards the specifically targeted language $L$ rather than multiple languages. As a result, we obtain $|L|$ multilingual many-to-one systems to serve all X-Y translation directions. We experiment with our proposed approach in the following two settings: 1) WMT’21 large-scale multilingual translation data with 972M sentence pairs and 2) our in-house production-scale dataset with 4.1B sentence pairs.

We experiment with two small tasks of the WMT’21 large-scale multilingual translation task. The tasks provide multilingual multi-way parallel corpora from the Flores 101 data Wenzek et al. (2021). The parallel sentences are provided among English (en), five Central and East European languages of {Croatian (hr), Hungarian (hu), Estonian (et), Serbian (sr), Macedonian (mk)} for the task 1, and five Southeast Asian languages of {Javanese (jv), Indonesian (id), Malay (ms), Tagalog (tl), Tamil (ta)} for the task 2. We removed sentence pairs either of whose sides is an empty line, and eventually collected the data with (English-centric, Non-English-centric)=(321M, 651M) sentence pairs in total. The data size per direction varies in a range of 0.07M-83.9M. To balance the data distribution across languages Kudugunta et al. (2019), we up-sample the low-resource languages with temperature=5. We append language ID tokens at the end of source sentences to specify a target language Johnson et al. (2017). We tokenize the data with the SentencePiece Kudo and Richardson (2018) and build a shared vocabulary with 64k tokens.

We train Transformer models Vaswani et al. (2017) consisting of a $m$-layer encoder and $n$-layer decoder with (hidden dim., ffn dim.) =(768, 3072) in a complete multilingual many-to-many fashion. We have two settings of ($m$, $n$) = (12, 6) for “12E6D” and (24, 12) for “24E12D” , to learn diverse multilingual dataset. The model parameters are optimized by using RAdam Liu et al. (2020) with an initial learning rate of 0.025, and warm-up steps of 10k and 30k for the 12E6D and 24E12D model training, respectively. The systems are pretrained on 64 V100 GPUs with a mini-batch size of 3072 tokens and gradient accumulation of 16. After the pretraining, the models are finetuned on a subset of X-$L$ training data. We finetune the model parameters gently on 8 V100 GPUs with the same mini-batch size, gradient accumulations, and optimizer with different learning rate scheduling of (init_lr, warm-up steps)=({1e-4, 1e-5, 1e-6}, 8k). The best checkpoints are selected based on development loss. The translations are obtained by a beam search decoding with a beam size of 4, unless otherwise stated.

Deploying a larger and larger model is not always feasible. We often have limitations in the computational resources at inference time, which leads to a trade-off problem between the performance and the decoding cost caused by the model architecture. In this section, we validate our proposed approach in an extremely large-scale data setting and also discuss how we can build lighter NMT models without the performance loss, while distilling the proposed MNMT systems Kim and Rush (2016). We briefly touch the following three topics of 1) multi-way multilingual data collection, 2) English-centric vs. multi-centric pretraining for X-Y translations, and 3) a lighter NMT model that addresses the trade-off issue between performance and latency. Then, we report the experimental results in the extremely large-scale setting.

This paper proposes a simple but effective two-stage training strategy for MNMT systems that serve arbitrary X-Y translations. To support translations across languages, we first pretrain a complete multilingual many-to-many model, then transfer the representations via finetuning the model in a many-to-one multilingual fashion. In the WMT’21 translation task, we experimentally showed that the proposed approach substantially improve translation accuracy for most X-Y directions against the strong conventional baselines of bilingual systems, pivot translation systems, and many-to-one multilingual systems. We also examined the proposed approach in the extremely large-scale setting, while addressing the practical questions such as multi-way parallel data collection, the usefulness of multilinguality during the pretraining and finetuning, and how to save the decoding cost, achieving the better X-Y quality.