AdMix: A Mixed Sample Data Augmentation Method for Neural Machine Translation

In this stage, we introduce an appropriate amount of discrete noises to obtain several augmented samples. Let $X\in\mathbb{R}^{s\times V}$, $Y\in\mathbb{R}^{t\times V}$ represent a source sequence of length $s$ and a target sequence of length $t$, respectively. $V$ denotes the size of vocabulary. Given a pair of training example $(X,Y)$ in training set, we perform word replacement (WR), word swapping (WS), word dropping (WD) operations on the source and target sentences to obtain $(X_{wr},Y_{wr})$, $(X_{ws},Y_{ws})$, $(X_{wd},Y_{wd})$ respectively. For example, we swap the positions of $x_{1}$ and $x_{3}$, replace $x_{1}$ with $x_{1}^{\prime}$, and remove $x_{2}$ in the sentence for WS, WR, WD respectively (see Figure LABEL:fig:fig1).

In practice, to ensure that the amount of noise is proportional to the sentence length $l$, we set a hyperparameter $\gamma$. For WS and WR operations, the number of changed words is $n=\gamma l$. For WD operation, we randomly remove each word in the sentence with probability $\gamma$.

After obtaining the augmented sentences, we first obtain their embedding sequences separately: $(\mathbf{E}\left[\mathbf{X}_{wr}\right],\mathbf{E}\left[\mathbf{Y}_{wr}\right])$, $(\mathbf{E}\left[\mathbf{X}_{ws}\right],\mathbf{E}\left[\mathbf{Y}_{ws}\right])$, $(\mathbf{E}\left[\mathbf{X}_{wd}\right],\mathbf{E}\left[\mathbf{Y}_{wd}\right])$. Inspired by Hendrycks et al. Hendrycks et al. (2020), we choose to use elementwise convex combinations to mix them and the coefficient vector is randomly sampled from a Dirichlet($\alpha$, . . . , $\alpha$) distribution. The word embedding of these augmented data after softly mixing is $(\mathbf{E}\left[\mathbf{X}_{ad}\right],\mathbf{E}\left[\mathbf{Y}_{ad}\right])$. Finally, we use a residual connection to combine the mixed samples with their original samples by sampling $m$ from a Beta($\beta$, $\beta$) distribution, and the word embedding of the final synthetic sentence is

where $\mathbf{E}\left[\mathbf{X}\right],\mathbf{E}\left[\mathbf{Y}\right]$ denote the word embedding at the source and target sides of original sentence pairs respectively.

The pseudocode of AdMix is shown in Algorithm 1. We couple this augmentation scheme with a loss that enforces a consistent embedding by the model across diverse augmentations of the same input sentences. Since the semantic meanings of the sentences are approximately preserved after AdMix operation, we can incorporate Jensen-Shannon divergence consistency loss into the training objective by encouraging the model to make similar predictions between original samples and synthetic samples. For this purpose, we minimize the Jensen-Shannon divergence among the posterior distributions of the original sample $(x,y)$ and its augmented variants $(x_{admix},y_{admix})$. The training objective can be written as:

where $\mathcal{L}_{ce}$ denotes the standard cross-entropy loss applied to the original samples, and $T$ refers to the length of the target sentence. $JS$ denotes Jensen-Shannon divergence. $p_{orig}$ and $p_{admix}$ are the probability distributions of model by respectively feeding the original sentence and the augmented sentence. $\lambda$ is a hyperparameter, which balances the importance between two loss function.

The Jensen-Shannon divergence can be understood to measure the average information that the sample reveals about the identity of the distribution from which it was sampled Hendrycks et al. (2020). This loss can be computed by

where $M=(p_{orig}+p_{admix})/2$. Although KL-divergence has been widely adopted as the divergence metric in previous works Miyato et al. (2017); Clark et al. (2018); Xie et al. (2020), it has been shown that the JS divergence loss can endow the model with more stability and consistency across a diverse set of inputs Kannan et al. (2018); Hendrycks et al. (2020). Therefore, we utilize the Jensen-Shannon (JS) divergence consistency loss as our loss function in this paper.

Datasets: For IWSLT14 German-English, following Edunov et al. Edunov et al. (2018), we apply the byte-pair encoding (BPE) Sennrich et al. (2016b) script to preprocess the training corpus with 10K joint operations, which consists of 0.16M sentence pairs. The validation set is split from the training set and the test set is the concatenation of tst2010, tst2011, tst2012, dev2010, and dev2012. For the Chinese-English translation task, the training set is the LDC corpus which contains 1.25M sentence pairs. The validation set is the NIST 06 dataset, and test sets are NIST 02, 03, 04, 05, 08. We apply BPE for Chinese and English respectively, and the merge operations are both 32k. For English-German translation, we use the WMT14 corpus consisting of 4.5M sentence pairs. The validation set is newstest2013 and the test set is newstest2014. We build a shared vocabulary of 32K sub-words using the BPE script.

We compare our approach with the following baselines:

• •

  Transformer: The vanilla Transformer model without any data augmentation. Vaswani et al. (2017);

• •

  Swap: Randomly choose several tokens in the sentence and swap their positions. Artetxe et al. (2017);

• •

  WordDrop: Randomly drop each token in the sentence with probability $\gamma$. Lample et al. (2018);

• •

  Switchout: Randomly replace tokens over the vocabulary by position. Wang et al. (2018);

• •

  SeqMix: Randomly sample two sentences from training dataset and softly combine them. Guo et al. (2020).

For Swap and WordDrop, we set the probability $\gamma$ = 0.15 of each word token to be swapped or replaced in the training phase. As for Switchout, both the source and target side temperature parameters are set to 1.0 by us. When adopting the SeqMix training strategy, we set $\beta$ of Beta($\beta$, $\beta$) distribution as 1.0, 1.0 and 0.1 for Zh-En, De-En and En-De respectively. For AdMix, we fix $\beta$ = 1.0 of the Beta($\beta$, $\beta$) distribution and $\alpha$ = 1.0 of the Dirichlet($\alpha$, . . . , $\alpha$) distribution in all experiments.

We show the BLEU scores of different methods for Zh-En, En-De, and De-En translation tasks in Table 1. We compare our method with vanilla Transformer and existing methods including Swap, WordDrop, Switchout, SeqMix. AdMix significantly outperforms the Transformer baseline on all tasks. Specifically, compared to the baseline system, AdMix delivers significant improvements 1.70, 0.96, and 2.67 points for the Zh-En, En-De, and De-En respectively. The experimental result also shows that our method is more effective on small datasets.

AdMix shows performance advantages over other data augmentation methods, outperforming the best-in-class SwitchOut by 0.8 points in Zh-En and SeqMix by 1.61 points in De-En. In particular, the superiority of AdMix over SeqMix Guo et al. (2020) verifies that we propose a more effective method to generate synthetic examples in NMT. Our approach consistently outperforms SeqMix Guo et al. (2020), yielding significant 1.03, 0.16, and 1.61 BLEU point gains on the Zh-En, En-De, and De-En translation tasks, respectively. Compared with the method of discrete noise methods (such as WordDrop Lample et al. (2018)), our method yields significant 1.39, 0.71, 2.07 BLEU point gains on the Zh-En, En-De, and De-En translation tasks respectively.

To investigate the importance of each component of AdMix, the BLEU scores from the ablation experiments on the German-English dataset are presented in Table 2. Firstly, we remove one of the three discrete transformation operations which include word replacement, word dropping, and word swapping. And the results appear that no matter which of the three discrete noises is removed, the empirical performance will be damaged. Especially, the BLEU score in the test set drops by 0.37 after removing word swapping. Therefore, it is important to introduce diversity through these three discrete transformation operations. And among these three operations, the word swapping operation can bring more diversity with the same noise levels. We also remove the residual connection to observe the BLEU score, which drops from 37.10 to 36.95 in the test set. The residual connection is effective probably because it preserves the semantic information of clean sentences, allowing a better correspondence between the source and target sentences.

Finally, We wonder how the BLEU scores would change by only applying AdMix on the source or target sentences. As shown in Table 2, the data augmentation strategies performed on either the source or target sentences are not as effective as on both sides simultaneously. Specially, when we employ AdMix only on the target sentences, the BLEU score drops 1.48 points in the test set. Nevertheless, both strategies have different degrees of improvement compared to the baseline system. The former brings 2.54 BLEU improvement and the latter yields 1.19 BLEU improvement in the test set.

We select different values of the hyperparameter $\lambda$ (ranging from 0.0 to 40.0) to investigate the importance of incorporating the Jensen-Shannon (JS) divergence consistency loss. The coefficient of the cross-entropy (CE) loss term is set as 1, and the JS divergence consistency loss term is controlled as the relative weight through the coefficient $\lambda$. The results of the BLEU scores on the German-English dataset are shown in the left panel of Figure 2(a). The performance of the model tends to increase and then decrease as the lambda increases, producing the best empirical results on the validation set when the value $\lambda$ is 10. Compared with the standard translation loss, the JS divergence loss can lead to performance improvements. Because it can approximate the original sample output distribution $p(\tilde{y}|x;y)$ and the synthetic sample output distribution $p(\tilde{y}|\hat{x};\hat{y})$ , i.e. to ensure consistency at the semantic level.

Another important hyperparameter with the AdMix approach is the noise fractions $\gamma$, where the fractions can be regarded as the magnitude of perturbations applied to the input sentence. As shown in the right panel of Figure 2(b), we apply various noise fractions for AdMix, including 0.1, 0.2, 0.3 and 0.4. It can be observed that the noise fractions of 0.1 shows the best translation performance of the obtained model. Using a larger noise fractions within a certain range can also obtain higher levels of performance than the baseline. The results show that the introduction of discrete noise in the training samples has a positive impact on improving the translation quality of the model, and AdMix can maintain a stable bleu score even at higher noise levels.

The noising-based data augmentation methods can not only improve the translation quality of the model, but also improve its robustness Miyato et al. (2017); Li et al. (2021). To test robustness on noisy inputs, similarly to Cheng et al. Cheng et al. (2020), we construct a noisy data set by randomly replacing a word in each sentence of the standard German-English validation set with a relevant alternative. The cosine similarity of word embeddings is used to determine the relevance of words. We repeat the process in an original sentence according to the number of operations where zero operation yields the original clean dataset.

As shown in Table 3, we can find that our training approach consistently outperforms all baseline methods on all the numbers of operations. This demonstrates that our approach has the ability to resist perturbations. The performance on Transformer baseline decreases rapidly as the number of operations increases. Although the performance of our approach also drops, we can see that our approach consistently exceeds baseline. After conducting three operations on the original validation set, the vanilla Transformer decreases from 35.82 to 29.75, while AdMix decreases from 38.28 to 32.58.

To explore the effect of combining our method with back-translation (BT), we sample 0.16M English sentences from the WMT13 english monolingual corpus²²2https://www.statmt.org/wmt13/training-monolingual-news-2007.tgz. We compare AdMix against BT on De-En translation task. As shown in Table 4, with using extra monolingual data, the back-translation approach yields a boost of 1.27 BLEU compared to the vanilla Transformer. But the gain delivered by BT is less significant than the gain delivered by AdMix. In addition, AdMix and back translation are not mutually exclusive, and we can apply AdMix on the pseudo-parallel sentences obtained by BT to further improve the BLEU score. When we combine BT with AdMix, it will yield 0.36 BLEU improvement than using AdMix alone.