Multilingual Mix: Example Interpolation Improves
Multilingual Neural Machine Translation

Multilingual modeling has been receiving increasing research attention over the past few years, arising from successful demonstrations of improved quality across a variety of tasks, languages and modalities (Lample and Conneau, 2019; Arivazhagan et al., 2019b; Conneau et al., 2021). The success of these models is typically ascribed to vocabulary sharing, parameter tying and implicit pivoting through dominant languages like English (Conneau et al., 2020). These conventional techniques are effective, but might not be exploiting the full potential of multilingual models to learn the underlying inductive bias: the learning signal from one language should benefit the quality of other languages Caruana (1997); Arivazhagan et al. (2019b).

Here we study two related issues that exist in the context of multilingual Neural Machine Translation (NMT) training Dong et al. (2015); Firat et al. (2016a); Johnson et al. (2017). First, NMT models Bahdanau et al. (2015); Vaswani et al. (2017) are trained with maximum likelihood estimation which has a strong tendency to overfit and even memorize observed training examples, particularly posing challenges for low resource languages Zhang et al. (2018). Second, training examples from distinct language pairs are separately fed into multilingual NMT models without any explicit instance-level sharing (with the exception of multi-source NMT Zoph and Knight (2016); Firat et al. (2016b)); as a consequence, given large enough capacity, the models have the liberty to map representations of different languages into distinct subspaces, limiting the extent of cross-lingual transfer.

In this work, we introduce multilingual crossover encoder-decoder (mXEncDec) to address these issues following the recent work on XEncDec Cheng et al. (2021) and mixup Zhang et al. (2018); Cheng et al. (2020); Guo et al. (2020). Inspired by chromosomal crossovers Rieger et al. (2012),  mXEncDec fuses two multilingual training examples to generate crossover examples inheriting the combinations of traits of different language pairs, which is capable of explicitly capturing cross-lingual signals compared to the standard training which mechanically combines multiple language pairs. mXEncDec has the following advantages:

1. 1.

   Enhancing the cross-lingual generalization. Thanks to crossover examples generated by fusing different language pairs, the multilingual NMT is encouraged to learn to transfer explicitly via more languages rather than implicitly via the predominant languages.

2. 2.

   Improving the model generalization and robustness. As vicinity examples around each example in the multilingual corpus (akin to Vicinal Risk Minimization Chapelle et al. (2001)), crossover examples produced by mXEncDec can enrich the support of the training distribution and lead to better generalization and robustness respectively on general and noisy inputs Zhang et al. (2018).

3. 3.

   Alleviating overfitting to low-resource languages. mXEncDec can increase the diversity of low-resource languages by fusing low-resource examples with others, instead of the simple duplication in the standard training.

In mXEncDec, we randomly pick up two training examples drawn from the multilingual training corpus and first interpolate their source sentences where we have to prudently deal with language tags. Then we leverage a mixture decoder to produce a virtual target sentence. To account for heavy data imbalance of each language pair, we propose a pairwise sampling strategy to adjust interpolation ratios between language pairs. We also propose to simplify the target interpolation to cope with noisy attention and fusions of dissimilar language pairs. Different from XEncDec fusing two heterogeneous tasks Cheng et al. (2021), we attempt to adapt it to deeply fuse different language pairs.

Experimental results on a large-scale WMT multilingual dataset show that mXEncDec yields improvements of $+1.13$ and $+0.47$ BLEU points averagely on xx-en and en-xx test sets over a vanilla multilingual Transformer model. We also evaluate our approaches on zero-shot translations and obtain up to $+5.53$ BLEU points over the baseline method, which corroborates the better transferabilty of multilingual models with our approaches. The more stable performance on noisy input text demonstrates the capability of our approach to improve the model robustness. To further explain the model behaviors at the representation level, qualitative and quantitative comparisons on representations manifest that our approach learns better multilingual representations, which indirectly explicates the BLEU improvements.

Multilingual Neural Machine Translation. NMT (Bahdanau et al., 2015; Vaswani et al., 2017) optimizes the conditional probability $P(\mathbf{y}|\mathbf{x};\bm{\theta})$ of translating a source-language sentence $\mathbf{x}$ into a target-language sentence $\mathbf{y}$. The encoder reads the source sentence $\mathbf{x}=x_{1},...,x_{I}$ as a sequence of word embeddings $e(\mathbf{x})$. The decoder acts as a conditional language model over embeddings $e(\mathbf{y})$ and the encoder outputs with a cross-attention mechanism (Bahdanau et al., 2015). For clarity, we denote the input and output in the decoder as $\mathbf{z}$ and $\mathbf{y}$, i.e., $\mathbf{z}=\langle s\rangle,y_{1},\cdots,y_{J-1}$ as a shifted copy of $\mathbf{y}$, where $\langle s\rangle$ is a sentence start token. Then the decoder generates $\mathbf{y}$ as $P(\mathbf{y}|\mathbf{x};\bm{\theta})=\prod_{j=1}^{J}P(y_{j}|\mathbf{z}_{\leq j},\mathbf{x};\bm{\theta}).$ The cross-attention matrix is denoted as $\mathbf{A}\in\mathbb{R}^{J\times I}$. NMT optimizes the parameters $\bm{\theta}$ by maximizing the likelihood of a parallel training set $\mathcal{D}$:

where $\ell$ is the cross entropy loss between the model prediction $f(\mathbf{x},\mathbf{y};\bm{\theta})$ and label vectors $v(\mathbf{y})$ for $\mathbf{y}$. $v(\mathbf{y})$ could be a sequence of one-hot vectors with smoothing in Transformer Vaswani et al. (2017).

Multilingual NMT extends NMT from the bilingual to the multilingual setting, in which it learns a one-to-many, many-to-one or many-to-many mapping from a set of languages to another set of languages Firat et al. (2016a); Johnson et al. (2017). More specifically, the multilingual NMT model is learned over parallel corpora $\mathcal{M}=\{\mathcal{D}^{l_{i}}\}_{i=1}^{L}$ where $L$ is the number of language pairs:

where all the parallel training sets are fed into the NMT model.

The crossover encoder combines embeddings of the two source sequences into a new sequence of embeddings:

where $\mathbf{m}=m_{1},\cdots,m_{|\tilde{\mathbf{x}}|}\in\{0,1\}^{|\tilde{\mathbf{x}}|}$ is sampled from a distribution or constructed according to a hyperparameter ratio $p$; e.g., $p=0.15$ means that $15\%$ of elements in $\mathbf{m}$ are $0$. $|\tilde{\mathbf{x}}|$ is the length of $\tilde{\mathbf{x}}$, which is equal to $\mathop{\rm max}(|\mathbf{x}|,|\mathbf{x}^{\prime}|)$.

On the crossover decoder side, a mixture conditional language model is employed for the generation of the virtual target sentence. The input embedding $e(\tilde{z}_{j})$ and output label $v(\tilde{y}_{j})$ for the decoder at the $j$-th position are calculated as:

where $\mathbf{t}=t_{1},...,t_{|\tilde{\mathbf{y}}|}\in[0,1]^{|\tilde{\mathbf{y}}|}\subset\mathbb{R}^{|\tilde{\mathbf{y}}|}$. In contrast to a common language model fed with a single word $y_{j-1}$ for predicting $y_{j}$ at the $j$-th position, the crossover decoder aims to generate an interpolated vector $v(\tilde{y}_{j})$ by averaging $v(y_{j})$ and $v(y^{\prime}_{j})$ with $t_{j}$, on condition that the current input embedding is also weighted on embeddings $e(y_{j-1})$ and $e(y^{\prime}_{j-1})$ with $t_{j-1}$. The weight vector $\mathbf{t}$ used for interpolating target inputs and labels is computed as:

where $\mathbf{A}$ and $\mathbf{A}^{\prime}$ are the alignment matrices for $(\mathbf{x},\mathbf{y})$ and $(\mathbf{x}^{\prime},\mathbf{y}^{\prime})$. In practice the cross-attention scores in the NMT model are utilized as an alternative noisy alignment matrix Garg et al. (2019).

The cross-entropy is utilized to compute the loss for XEncDec when feeding $e(\tilde{\mathbf{x}})$, $e(\tilde{\mathbf{z}})$ and $v(\tilde{\mathbf{y}})$ into the encoder-decoder model, denoted as:

In this work, we aim to leverage XEncDec to encourage multilingual NMT models to better exploit cross-lingual signals with crossover examples created by explicitly fusing different language pairs. We introduce its variant, called mXEncDec as shown in Figure 1, in which the parent examples could belong to either the same or different language pairs. The subsequent subsections discuss how to address new challenges of mXEncDec for multilingual NMT.

Data and Evaluation. We conduct experiments on the English-centric WMT multilingual dataset composed of $16$ languages (including English) and $30$ translation directions from past WMT evaluation campaigns before and on WMT’$19$ Barrault et al. (2019). The data distribution is highly skewed, varying from roughly 10k examples in En-Gu to roughly 60M examples in En-Cs. Two non-English test sets, Fr-De and De-Cs, are used to verify zero-shot translations. In addition, we also use multi-way test sets in FLORES-101 Goyal et al. (2021) to analyze the trained multilingual models¹¹1See data details in Table 6 in the appendix.

To mitigate the data imbalance in the WMT multilingual corpus, we follow Arivazhagan et al. (2019b) and adopt a temperature-based data sampling strategy to over-sample the low-resource languages where the temperature is set to $5$. We apply SentencePiece Kudo and Richardson (2018) to learn a vocabulary of $64k$ sub-words. We perform experiments in three settings: many-to-one, one-to-many and many-to-many translations. The $15$ test language pairs are cast into three groups according to their data size: High ($>10M$, 5 languages), Low ($<1M$, 7) and Medium ($>1M\&<10M$, 3). We report not only the average detokenized BLEU scores for each group as calculated by the SacreBLEU script Post (2018) but also winning ratio (WR) indicating the ratio of all the test sets on which our approach beats the baseline method.

Multilingual NMT has made tremendous progress in recent years Dong et al. (2015); Firat et al. (2016a); Johnson et al. (2017); Arivazhagan et al. (2019b); Fan et al. (2021). Recent research efforts to improve the generalization of multilingual models concentrate on enlarging the model capacity Huang et al. (2019); Zhang et al. (2020); Lepikhin et al. (2020), incorporating hundreds of languages Fan et al. (2021), pretraining multilingual models Liu et al. (2020), and introducing additional regularization constraints Arivazhagan et al. (2019a); Al-Shedivat and Parikh (2019); Yang et al. (2021). Our work is related to the last three ones in that they try to enable models to better transfer across languages by introducing an alignment loss to learn an interlingua Arivazhagan et al. (2019a) or imposing an agreement loss on translation equivalents Al-Shedivat and Parikh (2019); Yang et al. (2021). However, we propose to utilize mXEncDec to directly combine language pairs for better exploitation of cross-lingual signals.

Another related research line is data mixing. Since mixup Zhang et al. (2018); Yun et al. (2019) was proposed in computer vision, we have observed great success in NLP Guo et al. (2019); Cheng et al. (2020); Guo et al. (2020); Cheng et al. (2021). mXEncDec shares the commonality of combining example pairs as inspired by XEncDec Cheng et al. (2021). To the best of our knowledge, we are the first to fuse different language pairs to improve cross-lingual generalization and robustness for multilingual NMT.

We have presented mXEncDec to fuse different language pairs at instance level for multilingual NMT, which enables the model to better exploit cross-lingual signals. Experimental results on general, zero-shot and noisy test sets demonstrate that our approach can significantly improve the cross-lingual generalization, zero-shot transfer and robustness of multilingual NMT models. Representation analyses further confirms that our approach is capable of learning better multilingual representations, which coincides with improvements in BLEU. We plan to investigate whether this approach can improve the model generalization in a broader scope like domain generalization. We find that mXEncDec can easily achieve notable improvements for xx-en translations because they share an identical target language. However, there still exists huge headroom for en-xx translations. We plan to explore how to interpolate target languages more effectively, for example, possibly considering language similarity.