Improving Neural Machine Translation by Denoising Training

Main experiments in Tab. 1 are conducted on 6 translation datasets: IWSLT14 English$\leftrightarrow$German Nguyen et al. (2020), WMT16 English$\leftrightarrow$Romanian Gu et al. (2018), IWSLT21 English$\leftrightarrow$Swahili²²2https://iwslt.org/2021/low-resource, WMT14 English$\leftrightarrow$German Vaswani et al. (2017), WMT20 Japanese$\leftrightarrow$English³³3http://www.statmt.org/wmt20 and WMT17 Chinese$\leftrightarrow$English Hassan et al. (2018). The data sizes can be found in Tab. 1, ranging from 160K to 20M. Notably, Japanese$\leftrightarrow$English and Chinese$\leftrightarrow$English are two distant and high-resource language pairs. The monolingual data used for back translation in Tab. 3 is randomly sampled from publicly available News Crawl corpus⁴⁴4http://data.statmt.org/news-crawl/. We use same valid& test sets with previous works for fair comparison excepts IWSLT21 English$\leftrightarrow$Swahili, where we sample 5K/ 5K sentences from the training set as valid/ test sets. We preprocess all data besides Japanese$\leftrightarrow$English via BPE Sennrich et al. (2016) with 32K merge operations. For Japanese$\leftrightarrow$English, we filter the parallel data with Bicleaner Sánchez-Cartagena et al. (2018) and apply the SentencePiece Kudo and Richardson (2018) to generate 32K subwords.

We follow Lin et al. (2021) to collect eight English-centric multilingual language pairs from IWSLT14⁵⁵5https://wit3.fbk.eu/, including Farsi (Fa), Polish (Pl), Arabic (Ar), Hebrew (He), Dutch (Nl), German (De), Italian (It), and Spanish (Es). Following Tan et al. (2019), we apply BPE with 30K merge operations, and use over-sampling strategy to balance the training data distribution with temperature of $T=2$. The hyper-parameters of removing and replacing are set as ratio$=$0.1, and the nearby swapping as span$=$3 due to their better performance in our preliminary studies. For evaluation, we use tokenized BLEU Papineni et al. (2002) as the metric for bilingual tasks excepts English$\rightarrow$Chinese and Japanese$\leftrightarrow$English, where we report SacreBLEU Post (2018). The sign-test Collins et al. (2005) was utilized for statistical significance test.

We validated our proposed DoT on Transformer⁶⁶6https://github.com/pytorch/fairseq Vaswani et al. (2017). All bilingual tasks are trained on Transformer-Big except IWSLT14 En$\leftrightarrow$De and WMT16 En$\leftrightarrow$Ro (trained on Transformer-Base) because of their extremely small data size. For multilingual experiments, we closely follow previous work Wu et al. (2019) to adopt smaller⁷⁷7Transformer-Base with $d_{ff}=1024$ and $n_{head}=4$ Transformer-Base due to the small-scale data volume of the IWSLT multilingual dataset. For fair comparison, we set beam size and length penalty as 5 and 1.0 for all language pairs. Notably, our data-level approaches neither modify model structure nor add extra FLOPS, thus they are feasible to deploy on any frameworks (e.g. DynamicConv Wu et al. (2019) and non-autoregressive translation Gu et al. (2018); Ding et al. (2021b)) and other sequence-to-sequence tasks (e.g. grammar error correction and summarization Liu et al. (2021b)). We will explore them in the future works.

Someone may doubt that why not introduce more powerful pretrained language models, e.g. mBART Liu et al. (2020b), as the baseline. We politely argue that it is not fair to directly compare DoT with mBART in the main results because: 1) Our DoT consumes significantly less parameters compared to mBART, 49M~200M vs. 610M; and 2) Our DoT uses only the parallel data, while mBART uses TB-level text during pretraing. In addition, to show the effectiveness and efficiency of our DoT, we report the comparison results between mBART and DoT in §3.3 (see Tab. 4). We show that although mBART performs well on low-resource settings, however, it achieve worse performance on the high-resource settings.

For Transformer-Big models, we adopt large batch strategy Edunov et al. (2018) (i.e. 458K tokens/batch) to optimize the performance. The learning rate warms up to $1\times 10^{-7}$ for 10K steps, and then decays for 30K (data volumes range from 2M to 10M) / 50K (data volumes large than 10M) steps with the cosine schedule; For Transformer-Base, we empirically adopt 65K tokens/batch for small data sizes, e.g. IWSLT14 En$\rightarrow$De and WMT16 En$\rightarrow$Ro. The learning rate warms up to $1\times 10^{-7}$ for 4K steps, and then decays for 26K steps. For regularization, we tune the dropout rate from [0.1, 0.2, 0.3] based on validation performance, and apply weight decay with 0.01 and label smoothing with 0.1. We use Adam optimizer  (Kingma and Ba, 2015) to train models. We evaluate the performance on an ensemble of last 10 checkpoints to avoid stochasticity. All models were trained on NVIDIA DGX A100 cluster.

Someone may doubt that DoT heavily depends on how to properly set the early-stop steps. To dispel the doubt, we investigate whether our approach is robust to different early-stop steps. In preliminary experiments, we tried several simple fixed early-stop steps according to the size of training data (e.g. training 40K En-De and early stop at 10K/ 15K/ 20K, respectively). We found that both strategies achieve similar performances. Thus, we decide to choose a simple threshold (i.e. 1/3 of the total training steps) for better reproduction.

We experimented on 12 language directions, including IWSLT14 En$\leftrightarrow$De, WMT16 En$\leftrightarrow$Ro, IWSLT21 En$\leftrightarrow$Sw, WMT14 En$\leftrightarrow$De, WMT20 Ja$\leftrightarrow$En and WMT17 Zh$\leftrightarrow$En. Tab. 1 reports the experimental results, DoT achieves significant improvements over strong baseline Transformer in 8 out of 12 directions under the significance test $p<0.01$, and the other 3 directions also show promising performance under the significance test $p<0.05$, showing the effectiveness of our method across data scales.

We report the results of DoT on Ja$\leftrightarrow$En and Zh$\leftrightarrow$En language pairs, which belong to different language families (i.e. Japonic, Indo-European and Sino-Tibetan). As shown in Tab. 1, DoT significantly improves the translation quality in all cases. In particular, DoT achieves averaged +0.7 BLEU points improvement over the baselines, showing the effectiveness and universality of our method across language pairs.

Cross-lingual pretrained models are expected to learn better representations when given more languages Liu et al. (2020b). To verify this hypothesis, we report the multilingual results in Tab. 2. As seen, DoT consistently and significantly outperforms the strong baseline on all language pairs, confirming the effectiveness of DoT under multilingual MT scenario.

To illustrate the complementarity between DoT and related data manipulation works, we list one monolingual data-based approach: Tagged Back Translation (BT, Caswell et al. 2019b) combines the synthetic data generated with target-side monolingual data and parallel data, and four representative parallel data-based approaches: a) Competence based Curriculum Learning (CL, Platanios et al. 2019) trains samples with easy to hard order, where the difficulty metric is the competence; b) Knowledge Distillation (KD, Kim and Rush 2016) trains the model with sequence-level distilled parallel data; c) Data Diversification (DD, Nguyen et al. 2020) diversifies the data by applying KD and BT on parallel data; d) Bidirectional Training (BiT, Ding et al. 2021c) pretrains the model with bidirectional parallel data. As seen in Tab. 3, DoT is a plug-and-play strategy to yield further improvements.

Here we only provide the empirical evidence for the complementary between our pretraining approach and conventional data manipulation strategies, more insights like Liu et al. (2021a) investigated should be explored in the future.

Song et al. (2019); Liu et al. (2020b) reconstruct the abundant unlabeled data with sequence-to-sequence language models, offering the benefit for the low-resource translation tasks Cheng et al. (2021). However, such benefits are hard to enjoy in high-resource settings in part due to the catastrophic forgetting French (1999) and domain discrepancy between pretrain and finetune Gururangan et al. (2020); Anonymous (2021). Actually, DoT can be viewed as a simple in-domain pretraining strategy, where the pretraining corpora are exactly matched with the downstream machine translation task. We randomly sample different data scales to carefully compare the effects of pretrained mBART ⁸⁸8https://dl.fbaipublicfiles.com/fairseq/models/mbart/mbart.cc25.v2.tar.gz Liu et al. (2020b) and our proposed DoT. The mBART is used to initialize the downstream bilingual models with the officially given scripts⁹⁹9https://github.com/pytorch/fairseq/blob/main/examples/mbart/README.md. To maximize the effect of mBART, we grid-searched the “--update-freq”¹⁰¹⁰10Increasing the batch size during finetuning mBART for large scale datasets is essential to achieve better performance. from 2 to 16 to simulate different batch sizes. As for our DoT, we employ Transformer-Base for 100K and 1M data scales, and Transformer-Big for other larger datasets, i.e. 5M, 10M, and 20M. notably, under such experimental settings, our DoT consumes ~200M parameters for the high resource setting, while the mBART contains significantly more parameters, i.e. ~610M, for all settings.

Tab. 4 shows that although mBART achieves significant higher performance under extremely low-resource setting, e.g. 100K, it undermines the model performance under high-resource settings, e.g. $>=$10M, while our DoT strategy consistently outperforms “Random” setting. We attribute the better performance on resource-rich settings to the “in-domain pretrain”, which is consistent with the findings of Gururangan et al. (2020); Anonymous (2021).

Recall that we simply employ several straightforward noises, e.g. word drop, word mask and span shuffle, during sequence-to-sequence reconstruction, which are then serve as the self-supervision signals. Gururangan et al. (2020) show that task-relevant signals are important during finetuning the pretrained models. In machine translation, reconstructing the cross-lingual knowledge is intuitively more informative than randomly drop, mask, and shuffling. We therefore deliberately design a translation-relevant denoising signal to provide more insight for our DoT. Concretely, during word masking, we input the code-switched words rather than [MASK] token for the input, where we closely follow Yang et al. (2020) to achieve the code-switching. We found that task-relevant signals enable the model to achieve further improvement by 0.3 BLEU points, confirming our claim.