Improving Zero-Shot Multilingual Translation with
Universal Representations and Cross-Mappings

We denote the input sequence of symbols as $\mathbf{x}=(x_{1},\ldots,x_{nx})$ and the ground-truth sequence as $\mathbf{y}=(y_{1},\ldots,y_{ny})$. The transformer model is based on the encoder-decoder architecture. The encoder is composed of $\mathnormal{N}$ identical layers. Each layer has two sublayers. The first is a multi-head self-attention sublayer, and the second is a fully connected feed-forward network. Both of the sublayers are followed by a residual connection operation and a layer normalization operation. The input sequence $\mathbf{x}$ will be first converted to a sequence of vectors. Then, this sequence of vectors will be fed into the encoder, and the output of the $\mathnormal{N}$-th layer will be taken as source state sequences. We denote it as $\mathbf{H}_{\mathbf{x}}$. The decoder is also composed of $\mathnormal{N}$ identical layers. In addition to the same kind of two sublayers in each encoder layer, the cross-attention sublayer is inserted between them, which performs multi-head attention over the output of the encoder. We can get the predicted probability of the $k$-th target word conditioned by the source sentence and the $k-1$ previous target words. The model is optimized by minimizing a cross-entropy loss of the ground-truth sequence with teacher forcing:

$$\mathcal{L}_{CE}=-\frac{1}{n_{y}}\sum_{k=1}^{n_{y}}\log p(y_{k}|\mathbf{y}_{<k},\mathbf{x};\theta),$$

(1)

where $n_{y}$ is the length of the target sentence and $\theta$ denotes the model parameters.

We define $L=\{l_{1},\ldots,l_{M}\}$ where $L$ is a collection of $M$ languages involved in the training phase. Following Johnson et al. (2017), we share all the model parameters for all the languages. Following Liu et al. (2020), we add a particular language id token at the beginning of the source and target sentences, respectively, to indicate the language.

Earth Mover’s Distance Based on the optimal transport theory Villani (2009); Peyré et al. (2019), the earth mover’s distance (EMD) measures the minimum cost to transport the probability mass from one distribution to another distribution. Assuming that there are two probability distributions $\mu$ and $\mu^{\prime}$, that are defined as:

$$\begin{split}&\mu=\{(\mathbf{w}_{i},m_{i})\}_{i=1}^{n},\quad s.t.\sum_{i}m_{i}=1;\\ &\mu^{\prime}=\{(\mathbf{w}^{\prime}_{j},m^{\prime}_{j})\}_{j=1}^{n^{\prime}},\quad s.t.\sum_{j}m^{\prime}_{j}=1,\end{split}$$

(2)

where each data point $\mathbf{w}_{i}\in\mathbb{R}^{d}$ has a probability mass $m_{i}$ ($m_{i}>0$). There are $n$ data points in $\mu$. We define a cost function $c(\mathbf{w}_{i},\mathbf{w}^{\prime}_{j})$ that determines the cost of per unit between two points $\mathbf{w}_{i}$ and $\mathbf{w}^{\prime}_{i}$. Given above, the EMD is defined as:

$$\begin{split}\mathcal{D}(\mu,\mu^{\prime})&=\min_{\mathbf{T}\geq 0}\sum_{i,j}\mathbf{T}_{ij}c(\mathbf{w}_{i},\mathbf{w^{\prime}}_{j})\\ s.t.\quad&\sum_{j=1}^{n^{\prime}}\mathbf{T}_{ij}=m_{i},\forall i\in\{1,\ldots,n\},\\ &\sum_{i=1}^{n}\mathbf{T}_{ij}=m^{\prime}_{j},\forall j\in\{1,\ldots,n^{\prime}\}.\end{split}$$

(3)

$\mathbf{T}_{ij}$ denotes the mass transported from $\mu$ to $\mu^{\prime}$.

Given above, we can convert the state sequences to distributions:

$$\begin{split}&\mu_{\mathbf{x}}^{\mathbf{H}}=\{(\mathbf{h}_{i},\frac{|\mathbf{h}_{i}|}{\sum_{i}|\mathbf{h}_{i}|})\}_{i=1}^{nx},\\ &\mu_{\mathbf{y}}^{\mathbf{H}}=\{(\mathbf{h}^{\prime}_{j},\frac{|\mathbf{h}^{\prime}_{j}|}{\sum_{j}|\mathbf{h}^{\prime}_{j}|})\}_{j=1}^{ny}.\end{split}$$

(6)

Then, the SMD is formally defined as follows:

$$\begin{split}\mathcal{D}&(\mu_{\mathbf{x}}^{\mathbf{H}},\mu_{\mathbf{y}}^{\mathbf{H}})=\min_{\mathbf{T}\geq 0}\sum_{i,j}\mathbf{T}_{ij}c(\mathbf{h}_{i},\mathbf{h^{\prime}}_{j}),\\ s.t.\quad&\sum_{j=1}^{ny}\mathbf{T}_{ij}=\frac{|\mathbf{h}_{i}|}{\sum_{i}|\mathbf{h}_{i}|},\forall i\in\{1,\ldots,nx\},\\ &\sum_{i=1}^{nx}\mathbf{T}_{ij}=\frac{|\mathbf{h}^{\prime}_{j}|}{\sum_{j}|\mathbf{h}^{\prime}_{j}|},\forall j\in\{1,\ldots,ny\}.\end{split}$$

(7)

As illustrated before, we want decoder to make consistent predictions conditioned on the equivalent state sequences. Considering that the vector norm and direction both have impacts on the cross-attention results of decoder, we use the Euclidean distance as the cost function. We didn’t use the cosine similarity based metric, because it only considers the impact of vector direction. The proposed SMD is a fully unsupervised algorithm to align the contextual representations of the two semantic-equivalent sentences.

Theoretical Analysis In zero-shot translation, the decoder should map the semantic representations from different languages to the target language space, even if it has never seen the translation directions during training. This ability needs the model to make consistent predictions based on the semantic-equivalent sentences, whatever the input language is. To improve the prediction consistency of the model, we propose an agreement-based training method. Because the source sentence $\mathbf{x}$ and target sentence $\mathbf{y}$ are semantically equivalent, the probability of predicting any other sentence $\mathbf{z}$ based on them should be always equal theoretically, which is denoted as:

$$p(\mathbf{z}|\mathbf{x})=p(\mathbf{z}|\mathbf{y}).$$

(10)

Specifically, the predicted probabilities of the $k$-th target word conditioned by the first $k-1$ words of $\mathbf{z}$ and the source and target sentences is equal:

$$p(z_{k}|\mathbf{z}_{<k},\mathbf{x};\theta)=p(z_{k}|\mathbf{z}_{<k},\mathbf{y};\theta),$$

(11)

where $\theta$ denotes the model parameters. Optimizing Equation 11 can not only help the encoder produce universal semantic representations but also help the decoder map different source languages to the particular target language space indicated by $\mathbf{z}$.

The final loss consists of three parts, the cross entropy loss (Equation 1), the optimal transport loss based on SMD (Equation 9) and the KL divergence loss for the agreement-based training (Equation 13):

$$\mathcal{L}=\mathcal{L}_{CE}+\gamma_{1}|\mathbf{x}|\mathcal{L}_{OT}+\gamma_{2}\mathcal{L}_{AT}$$

(14)

where $\gamma_{1}$ and $\gamma_{2}$ are two hyperparameters that control the contributions of the two regularization loss terms. Since $\mathcal{L}_{OT}$ is calculated on the sentence-level and the other two losses are calculated on the token-level, we multiply the averaged sequence length $|\mathbf{x}|$ to $\mathcal{L}_{OT}$. Among these three losses, the first term dominates the parameter update of the model, and determines the model performance mostly. The latter two regularization loss terms only slightly modify the directions of the gradients. Because the first loss term does not depend on $\mathbf{H_{y}}$, we apply the stop-gradient operation to $\mathbf{H_{y}}$ (Figure 1), which means that the gradients will not pass through $\mathbf{H_{y}}$ to the encoder.

We conduct experiments on the following multilingual datasets: IWSLT17, PC-6, and OPUS-7. The brief statistics of the training set are in Table 1. We put more details in the appendix.

We use the Stanford word segmenter Tseng et al. (2005); Monroe et al. (2014) to segment Arabic and Chinese, and the Moses toolkit Koehn et al. (2007) to tokenize other languages. Besides, integrating operations of 32K is performed to learn BPE Sennrich et al. (2016).

We use the open-source toolkit called Fairseq-py Ott et al. (2019) as our Transformer system. We implement the following systems:

The following systems are implemented based on our method:

All the results (including the intermediate results of the ’PivT’ system) are generated with beam size = $5$ and length penalty $\alpha=0.6$. The translation quality is evaluated using the case-sensitive BLEU Papineni et al. (2002) with the SacreBLEU tool Post (2018). We report the tokenized BLEU for Arabic, char-based BLEU for Chinese, and detokenized BLEU for other languages¹¹1BLEU+case.mixed+numrefs.1+smooth.exp+
tok.{13a,none,zh}+version.1.5.1. The main results are shown in Table 2. We report the averaged BLEU with the same target language on the PC-6 and OPUS-7 dataset for display convenience, and the detailed results are in the appendix. The ’Ours’ system significantly improves over the ’ZS’ baseline system and outperforms other zero-shot-based systems on all datasets. The two proposed methods, OT and AT, can both help the model learn universal and cross mappings , so they both can improve the model performance independently. These two methods also complement each other and can further improve the performance when combined together. Besides, ’Ours’ system can even exceed the ’PivT’ system when the distant language pairs in the IWSLT-b or the low-resource language pairs in the PC-6 bring severe error accumulation problems. We also compare the training speed and put the results in the appendix.

To verify whether our method can better align different languages’ semantic space, we visualize each model’s encoder output with the IWSLT test sets. We first select three languages: Germany, Italian, and Dutch. Then we filter out the overlapped sentences of the three languages from the corresponding test sets and create a new three-way-parallel test set. Next, we feed all the sentences to the encoder of each model and average the encoder output to get the sentence representation. Last, we apply dimension reduction to the representation with t-SNE Van der Maaten and Hinton (2008). The visualization result in Figure 2(a) shows that the ’ZS’ system cannot align the three languages well, which partly confirms our assumption that the conventional MNMT cannot learn universal representations for all languages. As a contrast, the ’Ours’ system (d) can draw the representation closer and achieve comparative results as the ’CL’ system (c) without large amounts of negative instances to contrast. The visualization results confirm that our method can learn good universal representation for different languages.

To verify whether our method can help map the semantic representation from different languages to the same space of the target language, we inspect the prediction consistency of the models when the model is fed with synonymous sentences from different languages. Precisely, we measure the pair-wise BLEU on the above IWSLT three-way-parallel test set. We choose one language as the target language, e.g., German, and then translate the other two languages, e.g., Italian and Dutch, to the target language. After obtaining these two translation files, we use one file as the reference, the other as the translation to calculate the BLEU, and then we swap the role of these two files to calculate the BLEU again. We average the BLEU scores to get the pair-wise BLEU, and the results in Table 3 show that our method can achieve higher results, which proves that our method can improve the prediction consistency.

The zero-shot translation usually suffers from capturing spurious correlations in the supervised directions, which means that the model overfits the mapping relationship from the input language to the output language observed in the training set Gu et al. (2019). This problem often causes the off-target prediction phenomenon where the model generates translation in the wrong target languages. To check whether our method can alleviate this phenomenon, we use the $\mathrm{Langdetect}$ ²²2https://github.com/Mimino666/langdetect toolkit to identify the target language and calculate the prediction accuracy as $1-n_{off-target}/n_{total}$. We also compare our method with the ’SRA’ and ’CL’ methods. The results are shown in Table 4. The ’ZS’ baseline system can achieve high prediction accuracy on the IWSLT dataset, but the performance begin to decline as the amount of data becomes unbalanced and the languages become more unrelated. On all the datasets, our method achieves higher prediction accuracy and outperforms all the contrast methods. We can conclude from the results that our method can reduce the spurious correlation captured by the model.