WeChat Neural Machine Translation Systems for WMT21

Deeper and wider architectures are used this year since they show strong capacity as the number of parameters increases. In our experiments, we use multiple model configurations with 20/25-layer encoders for deeper models and the hidden size is set to 1024 for all models. Compared to our WMT20 models Meng et al. (2020), we also increase the decoder depth from 6 to 8 and 10 as we find that gives a certain improvement, but deeper depths give limited performance gains. For the wider models, we adopt 8/12/15 encoder layers and 1024/2048 for hidden size. The filter sizes of models are set from 8192 to 15000. Note that all the above model configurations are applied to the following variant models.

The Transformer Vaswani et al. (2017) with Pre-Norm Xiong et al. (2020) is a widely used architecture in machine translation. It is also our baseline model as its performance and training stability is better than the Post-Norm counterpart.

Recent studies Liu et al. (2020); Huang et al. (2020) show that the unstable training problem of Post-Norm Transformer can be mitigated by modifying initialization of the network and the successfully converged Post-Norm models generally outperform Pre-Norm counterparts. We adopt these initialization methods Huang et al. (2020) to our training flows to stabilize the training of deep Post-Norm Transformer. Our experiments have shown that the Post-Norm model has a good diversity compared to the Pre-Norm Model and slightly outperform the Pre-Norm Model. We will further analyze the model diversity of different variants in Sec. 5.1.

We also use Average Attention Transformer (AAN) Zhang et al. (2018) as we used last year to introduce more model diversity. In the Average Attention Transformer, a fast and straightforward average attention is utilized to replace the self-attention module in the decoder with almost no performance loss. The context representation $g_{i}$ for each input embedding is as follows:

where $y_{k}$ is the input embedding for step $k$ and $i$ is the current time step. $FFN(\cdot)$ denotes the position-wise feed-forward network proposed by Vaswani et al. (2017).

In our preliminary experiments, we observe that the Self-BLEU Zhu et al. (2018) scores between AAN and Transformer are lower than the scores between the Transformer with different configurations.

We further explore three weighting strategies to improve the modeling of history information from previous positions in AAN. Compared to the average weight across all positions, we try three methods including decreasing weights with position increasing, learnable weights and exponential weights. In our experiments, We observe exponential weights perform best among all these strategies. The exponential weights context representation $g_{i}$ is calculated as follows:

where $\alpha$ is a tuned parameter. In our previous experiments, we test different alpha, including 0.3, 0.5, and 0.7, on the valid set and we set the alpha to 0.7 in all subsequent experiments as it slightly outperform the others.

Our preliminary experiments show that the decoder structure is strongly related to the model diversity in the Transformer. Therefore, we propose to stack different types of decoder layers to derive different Transformer variants. As shown in Figure 1, we mainly adopt three Mixed-AAN Transformer architectures: a) Alternately mixing the standard self-attention layer and the average attention layer, b) Continuously stacking several average attention layers on the bottom layers and then stacking self-attention layers for the rest layers. c) Stacking both the self-attention layer and average attention layer at each layer and using their average sum to form the final hidden states (named as ‘dual attention layer’).

In the experiments, Mixed-AAN not only performs better but also shows strong diversity compared to the vanilla Transformer. With four Mixed-AAN models, we reach a better ensemble result than the result with ten models which consist of deeper and wider standard Transformer. We will further analyze the effects of different architectures from performance, diversity, and model ensemble in Sec. 5.1

In Multi-Head Attention, the different attention heads perform separate computations, which are then summed at the end. Talking-Heads Attention Shazeer et al. (2020) is a new variation that inserts two additional learned linear projection weights, $W_{l}$ and $W_{w}$, to transform the attention-logits and the attention scores respectively, moving information across attention heads. The calculation formula is as follows:

We adopt this method in both encoders and decoders to improve information interaction between attention heads. This approach shows the most remarkable diversity among all the above variants with only a slight performance loss.

We filter the bilingual training corpus with the following rules for most language pairs:

• •

  Normalize punctuation with Moses scripts except Japanese data.

• •

  Filter out the sentences longer than 100 words or exceed 40 characters in a single word.

• •

  Filter out the duplicated sentence pairs.

• •

  The word ratio between the source and the target words must not exceed 1:4 or 4:1.

• •

  Filter out the sentences where the fast-text result does not match the origin language.

• •

  Filter out the sentences that have invalid Unicode characters.

Besides these rules, we filter out sentence pairs in which Chinese sentence has English characters in En-Zh parallel data. The monolingual corpus is also filtered with the n-gram language model trained by the bilingual training data for each language. All the above rules are applied to synthetic parallel data.

In this section, we describe our techniques for constructing general domain synthetic data. The general domain synthetic data is generated via large-scale back-translation, forward-translation and knowledge distillation to enhance the models’ performance for all domains. Then, we exploit the iterative in-domain knowledge transfer Meng et al. (2020) in Sec 3.3, which transfers in-domain knowledge to the vast source-side monolingual corpus, and builds our in-domain synthetic data. In the following sections, we elaborate the above techniques in detail.

Since in-domain knowledge transfer Meng et al. (2020) delivered a massive performance boost last year, we still use this technique in En$\leftrightarrow$Ja and En$\to$De this year. It is not applied to En$\to$Zh because no significant improvement is observed. We guess the reason is that the in-domain finetuning in the En$\to$Zh direction does not bring a significant improvement compared to the other directions. And in-domain knowledge transfer is aiming at enhancing the effect of finetuning, so this does not have a noticeable effect in the English-Chinese direction.

We first use normal finetuning in Sec. 3.5 to equip our models with in-domain knowledge. Then, we ensemble these models to translate the source monolingual data into the target language. We use 4 models with different architectures and training data as our ensemble model. Next, we combine the source language sentences with the generated in-domain target language sentences as pseudo-parallel corpus. Afterwards, we retrain our models with both in-domain pseudo-parallel data and general domain synthetic data.

Once the pseudo-data is constructed, we further obtain diverse data by adding different noise. Compared to previous years’ WMT competitions, we implement a multi-level static noise approach for our pseudo corpus:

• •

  Token-level: Noise on every single subword after byte pair encoding.

• •

  Word-level: Noise on every single word before byte pair encoding.

• •

  Span-level: Noise on a continuous sequence of tokens before byte pair encoding.

The different granularities of noise make the data more diverse. The noise types are random replacement, random deletion and random permutation. We apply the three noise types in a parallel way for each sentence. The probability of enabling each of the three operations is 0.2.

Furthermore, an on-the-fly noise approach is applied to the synthetic data. By using on-the-fly noise, the model is trained with different noises in every epoch rather than all the same along this training stage.

A domain mismatch exists between the obtained system trained with large-scale general domain data and the target test set. To alleviate this mismatch, we finetune these convergent models on small scale in-domain data, which is widely used for domain adaption Luong and Manning (2015); Li et al. (2019). We take the previous test sets as in-domain data and extract documents that are originally created in the source language for each translation direction Sun et al. (2019). We also explore several advanced finetuning approaches to strengthen the effects of domain adaption and ease the exposure bias issue, which is more serious under domain shift.

After we get numerous finetuned models, we need to search for the best combination for ensemble model. Ordinary random or greedy search is oversimplified to search for a good model combination and enumerate over all combinations of candidate models is inefficient. The Self-BLEU based pruning strategy Meng et al. (2020) we proposed in last year’s competition achieve definite improvements over the ordinary ensemble.

However, diversity is not the only feature we need to consider but the performance in the valid set is also an important metric. Therefore, we combine Self-BLEU and valid set BLEU together to derive a Boosted Self-BLEU-based Ensemble (BSBE) algorithm. Then, we apply a greedy search strategy in the top N ranked models to find the best ensemble models.

See algorithm 1 for the pseudo-code. The algorithm takes as input a list of n strong single models M, BLEU scores on valid set for each model B, average Self-BLEU scores for each model S, the number of models $n$ and the number of ensemble models $c$. The algorithm return a list C consists of selected models. We calculate the weighted score for each model as line 2 in the pseudo-code. The weight calculated in line 3 is a factor to balance the scale of Self-BLUE and valid set BLEU. Then the list C initially contains the model $m_{top}$ has a highest weighted score. Next, we iteratively re-compute the average Self-BLEU between the remaining models in $|M-C|$ and selected models in C, based on which we select the model has minimum Self-BLEU score into C.

In our experiments, we save around 95% searching time by using this novel method to achieve the same BLEU score of the Brute Force search. We will further analyze the effect of Boosted Self-BLEU based Ensemble in section 5.2.

The implementation of our models is based on Fairseq¹¹1https://github.com/pytorch/fairseq for En$\to$Zh and EN$\to$De, and OpenNMT²²2https://github.com/OpenNMT/OpenNMT-py for En$\leftrightarrow$Ja. All the single models are carried out on 8 NVIDIA V100 GPUs, each of which has 32 GB memory. We use the Adam optimizer with $\beta_{1}$ = 0.9, $\beta_{2}$ = 0.998. The gradient accumulation is used due to the high GPU memory consumption. The batch size is set to 8192 tokens per GPU and we set the “update-freq” parameter in Fairseq to 2. The learning rate is set to 0.0005 for Fairseq and 2.0 for OpenNMT. We use warmup step = 4000. We calculate sacreBLEU³³3https://github.com/mjpost/sacrebleu score for all experiments which is officially recommended.

The statistics of all training data is shown in Table 1. For each language pair, the bilingual data is the combination of all parallel data released by WMT21. For monolingual data, we select data from News Crawl, Common Crawl and Extended Common Crawl, it is then divided into several parts, each containing 50M sentences.

For general domain synthetic data, we use all the target monolingual data to generate back-translation data and a part of source monolingual data (about 80 to 100 million for different languages) to get forward translation data. For the in-domain pseudo-parallel data, we use the entire source monolingual data and bilingual data. All the test and valid data from previous years are used as in-domain data.

We use the methods described in Sec. 3.1 to filter bilingual and monolingual data.

English and German sentences are segmented by Moses⁴⁴4http://www.statmt.org/moses/, while Japanese use Mecab⁵⁵5https://github.com/taku910/mecab for segmentation. We segment the Chinese sentences with an in-house word segmentation tool. We apply punctuation normalization in English, German and Chinese data. Truecasing is applied to English$\leftrightarrow$ Japanese and English$\to$German. We use byte pair encoding BPE Sennrich et al. (2016b) with 32K operations for all the languages.

For the post-processing, we apply de-truecaseing and de-tokenizing on the English and German translations with the scripts provided in Moses. For the Chinese translations, we transpose the punctuations to the Chinese format.

The results of En$\to$Zh on newstest2020 are shown in Table 2. For the En$\to$Zh task, filtering out part of sentence pairs containing English characters in Chinese sentences shows a significant improvement in the valid set. After applying large-scale Back-Translation, we obtain +2.0 BLEU score on the baseline. We further gain +0.62 BLEU score after applying knowledge distillation and +0.24 BLEU from Forward-Translation. Surprisingly, we observe that adding more BT data from different shards with different generation strategy can further boost the model performance to 48.17. The finetuned model achieves a 49.81 BLEU score, which demonstrates that the domain of the training corpus is apart from the test set domain. The advanced finetuning further brings about 0.41 BLEU score gains compared to normal finetune. Our best single model achieves a 50.22 BLEU score.

In preliminary experiments, we select the best performing models as our ensemble combinations obtaining +0.4 BLEU score. On top of that, even after searching hundreds of models, no better results are obtained. With BSBE strategies in Sec. 3.6, a better model combination with less number of models are quickly searched, and we finally achieve 50.94 BLEU score. Our WMT2021 English$\to$Chinese submission achieves a SacreBLEU score of 36.9, which is the highest among all submissions and chrF score of 0.337.

The results of En$\to$Ja on newstest2020 are shown in Table 2. For the En$\to$Ja task, we filter out the sentence pairs containing Japanese characters in the English side and vice versa. The Back-Translation and Knowledge Distillation improve the baseline from 35.78 to 36.66. Adding more BT data further brings in 0.56 improvements. The improvement by finetuning is much larger than other directions, which is 5.32 BLEU. We speculate that this is because there is less bilingual data for English and Japanese than for other languages, and the test results for Japanese are char level BLEU so this direction is more influenced by the in-domain finetuning. Two In-domain knowledge transfers improve BLEU score from 37.22 to 43.69. Normal finetune still provides 0.54 improvements after in-domain knowledge transfer. Then, we apply advanced finetuning methods to further get 0.19 BLEU improvements. Our final ensemble result outperforms baseline 9.57 BLEU.

The Ja$\to$En task follows the same training procedure as En$\to$Ja. From Table 2, we can observe that Back-Translation can provide 1.11 BLEU improvements from baseline. Knowledge Distillation and more BT data can improve the BLEU score from 20.82 to 22.11. The finetuning improvement is 3.8 which is slightly less than the En$\to$Ja direction but still larger than En$\to$Zh and En$\to$De. We also apply two-turn in-domain knowledge transfer and further boost the BLEU score to 25.89. After normal finetuning, the BLEU score achieves 26.27. The advanced finetuning methods provide a slight improvement on Ja$\to$En. After ensemble, we achieve 28.24 BLEU in newstest2020.

The results of En$\to$De on newstest2020 are shown in Table 2. After adding back-translation, we improve the BLEU score from 33.28 to 35.28. Knowledge Distillation further boosts the BLEU score to 36.58. The finetuning further brings in 2.63 improvements. After injecting the in-domain knowledge into the monolingual corpus, we get another 0.31 BLEU gain. We apply a post-processing procedure on En$\to$De. Specifically, we normalize the English quotations to German ones in German hypotheses, which brings in 1.3 BLEU improvements.

We conduct several experiments to validate the effectiveness of Transformer Vaswani et al. (2017) variants we used and list results in Table 3. We also investigate the diversity of different variants and the impacts on the model ensemble. The results is listed in Table 4 and Table 5. Here we take En$\to$Zh models as examples to conduct the diversity and ensemble experiments. The results in other directions show similar trends.

To verify the superiority of our Boosted Self-BLEU based Ensemble (BSBE) method, we randomly select 10 models with different architecture and training data. For our submitted system, we search from over 500 models. We use a greedy search algorithm Deng et al. (2018) as our baseline. The greedy search greedily selects the best performance model into candidate ensemble models. If the selected model provides a positive improvement, we keep it in the candidates. Otherwise, it is added to a temporary model list and still has a weak chance to be reused in the future. One model from the temporary list can be reused once, after which it is withdrawn definitely. We compare the results of greedy search, BSBE and Brute Force and list the ensemble model BLEU and the number of searches in Table 6. Note that $n$ is the number of models, which is 10 here. For BSBE, we need to get the translation result of every model to calculate the Self-BLEU. After that, we only need to perform the inference process once.

In this section, we describe our experiments on advanced finetuning in the four translation directions. As shown in Table 7, all the advanced finetuning methods outperform normal finetuning. For En$\to$Zh, Scheldule Sampling Based on Decoding Steps with Graduated Label Smoothing improves the model performance from 49.81 to 50.20. For En$\leftrightarrow$Ja, Target Denoising with Graduated Label Smoothing provides the highest BLEU gain, which are 0.19 and 0.11. For the En$\to$De direction, Confidence-Aware Scheldule Sampling with Graduated Label Smoothing performs the best, improving from 39.21 to 39.42. These findings are in line with the conclusion of Wang and Sennrich (2020) that links exposure bias with domain shift.