Refining Low-Resource Unsupervised Translation by Language Disentanglement of Multilingual Model

While our method starts with finetuning a fully parameter-shared multilingual MT model, we desire to gradually and progressively increase the level of language separation by reducing the degree of parameter sharing of the model. Thus, for every layer $j$ of an $H$-layer Transformer decoder such that $j$ is divisible by a constant $\sigma$ (e.g., 3) and $1\leq j\leq H$, we propose to replace its vanilla feed-forward layer ($\text{FFN}_{j}$) with $N$ separate layers $\overline{\text{FFN}}_{j,i}$ such that each corresponds to a language $l_{i}\in{\mathcal{L}}$.

During the training process, each of the separate $\overline{\text{FFN}}_{j,i}$ layers is sharded and reside individually on a distinct GPU-$i$ accelerator, where their back-propagated gradients are not aggregated or averaged during the data parallelism process. Each GPU-$i$ device is only fed with data from its respective language $l_{i}$ and the common language English. Meanwhile, the remaining parameters (e.g., embedding and attention layers) are still shared and their gradients are averaged across all GPU devices during back-propagation. Figure 1 visualizes a sharded FFN layer of our model and how its parameters are allocated on each GPU. During inference, $\overline{\text{FFN}}_{j,i}$ is used to decode the translation to and from $l_{i}$.

Formally, let $\hat{\theta}=\{\hat{\theta}_{m},\hat{\theta}_{f}\}$ be a fully parameter-shared multilingual model (e.g., CRISS) at our initialization, where $\hat{\theta}_{m}$ is the set of parameters intended to be shared and $\hat{\theta}_{f}$ is the initial set of FFN parameters intended to be disentangled in our proposed model. We define our model as $\theta=\{\theta_{m},\theta_{f}^{1},\theta_{f}^{2},...,\theta_{f}^{N}\}$, where $\theta_{m}$ denotes the shared parameters while $\theta_{f}^{i}$ denotes the separate parameters of all $\overline{\text{FFN}}_{j,i}$ layers ($1\leq j\leq H$) for a language $l_{i}\in{\mathcal{L}}$. Before we begin training the model, we initialize $\theta_{m}=\hat{\theta}_{m}$ and $\theta_{f}^{i}=\hat{\theta}_{f},\leavevmode\nobreak\ \forall i\in[1,...,N]$. Then during each update step, the gradients of $\theta_{f}^{i}$ and $\theta_{m}$ are computed as follow:

$$\nabla_{\theta}=\{\nabla_{\theta_{m}},\nabla_{\theta_{f}^{1}},\ldots,\nabla_{\theta_{f}^{N}}\}$$

(1)

$$\nabla_{\theta_{f}^{i}}=\mathop{\mathbb{E}}_{\begin{subarray}{c}x_{i}\sim{\mathbb{X}}_{i}\\ x_{e}\sim{\mathbb{X}}_{e}\end{subarray}}\nabla_{\theta_{f}^{i}}\big{(}{\mathcal{J}}_{\theta}(x_{i}|y_{i}^{e})+{\mathcal{J}}_{\theta}(x_{e}|y_{e}^{i})\big{)}$$

(2)

$$\nabla_{\theta_{m}}=\frac{1}{N}\sum_{l_{i}\in{\mathcal{L}}}\mathop{\mathbb{E}}_{\begin{subarray}{c}x_{i}\sim{\mathbb{X}}_{i}\\ x_{e}\sim{\mathbb{X}}_{e}\end{subarray}}\nabla_{\theta_{m}}\big{(}{\mathcal{J}}_{\theta}(x_{i}|y_{i}^{e})+{\mathcal{J}}_{\theta}(x_{e}|y_{e}^{i})\big{)}$$

(3)

where ${\mathbb{X}}_{i}$ and ${\mathbb{X}}_{e}$ are monolingual corpora (or data distribution) of language $l_{i}$ and English respectively, ${\mathcal{J}}_{\theta}(x|y)=-\log P_{\theta}(x|y)$, and $y_{i}^{e}\sim P(\cdot|x_{i},\{\theta_{m},\theta_{f}^{i}\})$ is the back-translation into English from $x_{i}$ while $y_{e}^{i}\sim P(\cdot|x_{e},\{\theta_{m},\theta_{f}^{i}\})$ is the back-translation into $l_{i}$ from English by the model $\{\theta_{m},\theta_{f}^{i}\}$. Equations 2 and 3 show that gradients are not aggregated for the language-specific sharded FFN layers, while the remaining are aggregated and averaged across all data streams. However, note that our model only separates the FFN layers of the decoder, while all encoder parameters are fully shared to ensure the language-agnostic representations from the encoder [31, 23].

With the aforementioned language-specific FFN layers, we enter the first stage of refinement. The purpose of this stage is to begin differentiating the language-specific FFN layers so that each FFN can specialize in its designated language. That is, after initializing our model $\theta=\{\theta_{m},\theta_{f}^{1},...,\theta_{f}^{N}\}$ with the pre-trained multilingual UMT model $\hat{\theta}=\{\hat{\theta}_{m},\hat{\theta}_{f}\}$ as $\theta_{m}=\hat{\theta}_{m}$ and $\theta_{f}^{i}=\hat{\theta}_{f}\hskip 5.0pt\forall i\in[1,...,N]$, we finetune it with iterative back-translation [9], except with some noteworthy modifications. Specifically, we uniformly sample monolingual sentences $x_{e}$ and $x_{i}$ from English and each low-resource language $l_{i}\in{\mathcal{L}}$, respectively. Then, we use $\theta_{i}=\{\theta_{m},\theta_{f}^{i}\}$, a subset of $\theta$, to back-translate $x_{e}$ and $x_{i}$ into $y_{e}^{i}$ and $y_{i}^{e}$, respectively. After that, we train the model with $y_{e}^{i}\rightarrow x_{e}$ and $y_{i}^{e}\rightarrow x_{i}$ pairs. Note that for each direction of back-translation, forward and backward directions, the appropriate FFN layers $\theta_{f}^{i}$ are used with respect to the data stream involving language $l_{i}$ and English on a separate GPU-$i$. Furthermore, similarly to Conneau and Lample [9], the back-translation model is continuously updated after each training step. The resulting model $\theta^{(1)}=\{\theta_{m}^{(1)},\theta_{f}^{1,(1)},...,\theta_{f}^{N,(1)}\}$ will be used to initialize the second stage of refinement.

Alleviating the curse of multilinguality may involve separation of unrelated languages. Yet, for many low-resource translation tasks, English is often significantly distant from the target low-resource counterparts, such as Nepali and Sinhala, which in fact share certain similarities between themselves. Thus, English may not share similar structural patterns with any of the target languages [32]. Despite that, most existing UMT models are bidirectional [9, 30, 22]. This means that they have to handle both translations to English and their target low-resource language $l_{i}$, causing them to endure reordering discrepancy and perform sub-optimally for both directions.

The second stage is aimed to disentangle English. It builds two separate models $\theta^{(2)}_{\rightarrow{\mathcal{L}}}$ and $\theta^{(2)}_{\rightarrow En}$ from $\theta^{(1)}$, which are specialised to translate to target low-resource languages $l_{i}\in{\mathcal{L}}$ and to English, respectively. Specifically, the second stage starts by initializing both $\theta^{(2)}_{\rightarrow{\mathcal{L}}}$ and $\theta^{(2)}_{\rightarrow En}$ with $\theta^{(1)}$. We then finetune them with iterative back-translation exclusively in $En\rightarrow l_{i}$ and $l_{i}\rightarrow En$ directions, respectively. Different from the first stage, we use a fixed model $\theta^{(1)}$ to back-translate the monolingual data. In other words, to finetune $\theta^{(2)}_{\rightarrow{\mathcal{L}}}$, we first initialize it with $\theta^{(1)}$. Then we sample monolingual data $x_{i}$ uniformly from all $l_{i}\in{\mathcal{L}}$ and use the fixed model $\theta^{(1)}$ to back-translate them into English $y_{i}^{e}\sim P(\cdot|x_{i},\{\theta_{m}^{(1)},\theta_{f}^{i,(1)}\})$. The resulting pair $y_{i}^{e}\rightarrow x_{i}$ is used to train $\theta^{(2)}_{\rightarrow{\mathcal{L}}}$. In the opposite direction, we finetune $\theta^{(2)}_{\rightarrow En}$ by first initializing it with $\theta^{(1)}$, then sample English data $x_{e}$ and randomly choose an $l_{i}\in{\mathcal{L}}$ to back-translate $x_{e}$ into $y_{e}^{i}\sim P(\cdot|x_{e},\{\theta_{m}^{(1)},\theta_{f}^{i,(1)}\})$, and finally train $\theta^{(2)}_{\rightarrow En}$ with $y_{e}^{i}\rightarrow x_{e}$ pairs.

The third stage improves the performance further by continuing finetuning $\theta^{(2)}_{\rightarrow{\mathcal{L}}}$ and $\theta^{(2)}_{\rightarrow En}$ into $\theta^{(3)}_{\rightarrow{\mathcal{L}}}$ and $\theta^{(3)}_{\rightarrow En}$, respectively. Specifically, we build $\theta^{(3)}_{\rightarrow{\mathcal{L}}}$ by initializing it with $\theta^{(2)}_{\rightarrow{\mathcal{L}}}$ and use $\theta^{(2)}_{\rightarrow En}$ as a fixed back-translation model and train $\theta^{(3)}_{\rightarrow{\mathcal{L}}}$ in the same manner as in section 3.3. Similarly, $\theta^{(3)}_{\rightarrow En}$ is trained in the same fashion in the opposite direction.

The last stage is aimed to achieve the maximal degree of language separation not only from English, but also between all languages in the target group. Specifically, we seek to train $N$ separate models $\theta^{(4)}_{\rightarrow l_{i}}$ for each language $l_{i}\in{\mathcal{L}}$. Each model $\theta^{(4)}_{\rightarrow l_{i}}$ is initialized with $\theta^{(3)}_{\rightarrow{\mathcal{L}}}$ with the corresponding language-specific FFN layers, i.e., $\{\theta_{m}^{(3)},\theta_{f}^{i,(3)}\}_{\rightarrow{\mathcal{L}}}$, and $\theta^{(4)}_{\rightarrow l_{i}}$ is trained by back-translation with a fixed backward model $\theta^{(3)}_{\rightarrow En}$, and exclusively on $En\rightarrow l_{i}$ direction. Because only one direction is trained in this stage, we no longer use the above language-specific FFN layers for our end models. Instead, we revert back to the vanilla Transformer, whose FFN layers are initialized with the appropriate language-specific layers from the previous stage. Note that we do not perform this stage of refinement for the to-English ($\rightarrow En$) direction, due to the fact that the encoder is fully shared across all languages and there is no difference in separating $l_{i}\rightarrow En$ directions for the encoder, while the decoder only decodes the common English language.

In this section, we conduct a comprehensive ablation study to gain more insights of our method and the importance of its components. In particular, we examine the contribution of each stage of our procedure to the overall performance gain across various low-resource languages (Table 2). We also investigate how our decoder language-specific FFN layers perform in comparison to other possible variants, such as language-specific attention layers.

In this section, we compare our refinement procedure against other related methods, as well as those that share a similar language disentanglement objective as ours. Particularly, we reproduce the method proposed by Sen et al. [27] in low-resource unsupervised MT. This method is equivalent to stage 1 of our refinement procedure, except that the entire decoder is linguistically disentangled with separate decoders for different languages, not just FFN parameters. Meanwhile, Sachan and Neubig [26] and Zhang et al. [34] both share the same goal of alleviating the curse of multilinguality, but in the context of supervised multilingual translation, where the model teaches itself to implicitly “select” relevant language-specific layers by gating mechanisms or gradient readjustments. We adapt these methods to unsupervised MT by substituting supervised training data with multilingual back-translated samples by the models. Lastly, Zuo et al. [35] recently propose a mixture-of-experts (MoE) variant [18, 29], where experts (FFNs) are selected randomly to perform the forward pass. In the context of language disentanglement, these separate experts act as implicit language-specific components. We also adapt this method to low-resource unsupervised tasks with multilingual back-translation and set the number of experts as the number of languages. For all methods, we use CRISS as the initial model.

The comparison results are shown in Table 4. As shown, the method proposed by Sen et al. [27] performs slightly above stage 1 of our method, while it underperforms our complete procedure by up to 2.7 BLEU in Gu-En task. This is in line with the fact that it is almost equivalent to our first stage. The approaches suggested by Sachan and Neubig [26] and Zhang et al. [34], on the other hand, yield considerably lower BLEU scores in various low-resource unsupervised tasks. A possible reason for this underperformance is that these methods are designed for supervised multilingual translation, where accurate parallel data plays a crucial role. When adapting to the unsupervised regime, the synthetic training samples created by back-translation may be too noisy and inaccurate. Meanwhile, the MoE candidate [35] also produce lower BLEU scores than our method, which can be due to its lack of explicit language-specific components to enforce a language disentanglement naturally.

Apart from iterative back-translation, there are other techniques that have been shown to improve unsupervised machine translation. One of them is cross-model back-translated distillation [22], or CBD, where two distinct UMT teachers are used to distill the final model. Another is pseudo-parallel data mining [31, 23], where language-agnostic representations are built for sentences across different languages, which are then used to map unlabeled sentences from one language to another to create synthetic parallel data [2]. CRISS is also itself pseudo-parallel data mining method. In this segment, we demonstrate that the aforementioned techniques do not adapt well with low-resource unsupervised tasks, which shifts our attention to other areas, like the curse of multilinguality. Specifically, we apply CBD on CRISS by finetuning two distinct models from pre-trained CRISS and using them to distill a final model. For pseudo-parallel mining, we use LASER [2] to mine pseudo-parallel data in only the low-resource directions of interest using the encoder outputs from CRISS itself.

The results are reported in Table 5. As shown, CBD only performs relatively similar to our stage 1. This indicates that the CBD only improves due to back-translation, and not its own distillation effect. Nonetheless, CBD faces a disadvantage that the two teachers are not considerably distinct as they are both finetuned from the same pre-trained CRISS model, which is not advisable in the original paper. This is nonideal as there is no existing well-performing pre-trained model other than CRISS.

Meanwhile, the mined pseudo-parallel data contributes little to the performance improvement due to the reality that the amount and quality of mined data is too low for low-resource pairs, which is less than 5000 pairs for En-Ne and En-Si. This forces the model to be trained on such small mined datasets by progressively decreasing loss weight to prevent overfitting. While many may attribute this to the model’s failure in mining more high-quality pseudo-parallel data, we empirically show that it is more likely that there are in fact not enough real parallel data in low-resource corpora to be mined! Specifically, we use our outperfoming model to back-translate the entire English monolingual corpus to Nepali. For each resulting back-translated Ne sentence, we search for its closest real Ne sentence by token-based Levenshtein distance [19] in the Ne monolingual corpus, and then filter out pairs whose distance is more than 20% the sentence length. We repeat the process for En-Si as well. As shown in Table 6, out of >100M possible unfiltered pairs, only <6000 samples satisfy the filtering criterion. We provide an in-depth analysis on this issue in the Appendix.