High-Quality Data Augmentation for Low-Resource NMT: Combining a Translation Memory, a GAN Generator, and Filtering

We employ the retrieval method to find sentences $s_{t}$ in the TM that are semantically similar to the source input $s$, along with their corresponding target sentences $t_{t}$. Here, we introduce a threshold for the Euclidean distance between sentence vectors to ensure the quality of the retrieved TM. To ensure the quality of the retrieved synthetic sentences that will be introduced in Section 2.2, we will discuss how to control generation in Section 2.2 and how to filter high qulaity sentences in Section 2.3. Additionally, considering the limited number of sentences in the low-resource language TM, we avoid setting this threshold too high or too low. In subsequent experiments, the threshold is set to 0.5 [22].

Sentence pairs $(s_{t},t_{t})$ with a distance below this threshold will be concatenated with $s$ using the following format: ’[SEP] $s$ [SEP] $s_{t}$ [SEP] $t_{t}$’, where [SEP] denote separators. The process of integrating TM into the input is presented in Figure 1.

Firstly, we input each sentence in the TM into a pre-trained sentence embedding model [14] to generate sentence vectors. Subsequently, we calculate the similarity score between two sentences by computing the Euclidean distance between their respective sentence vectors.

To facilitate rapid retrieval between the input vector representation and the corresponding vector of sentences in the TM, we employ FAISS toolkit [8].

In this paper, integrating the findings of previous work and the current research landscape of NMT, we use a vanilla Transformer as a generator, $G$, for complex translation tasks, to generate synthetic data in the target language, i.e., the low-resource language in our settings. To build this generator $G$, we design a fusion neural network as a discriminator $D$, composed of parallel BiLSTM and CNN, handling a binary classification task. As [4, 21, 25] explained, $D$ and $G$ play the following two-player minimax game with value function $V(G,D)$:

In the Equation (2), $(x,y)$ is a sentence pair from the bilingual corpus, $(x_{1},y_{1}^{\prime})$ is a source sentence in the monolingual corpus combined with the target sentence generated by $G$. $P_{data}$ represents the real data distribution and $P_{G}$ denotes the generator distribution. In this way, $D$ continually learns how to distinguish real and natural sentences and not so natural sentences generated by $G$. Meanwhile, $G$ strives to produce the most natural sentences to deceive $D$.

On the generator side, two loss functions are employed to improve performance. Since this is essentially still a translation task, we use the original loss function of the translation task. This differs somewhat from a classical GAN model. The loss value is derived from training the generator using the bilingual corpus. Additionally, we feed the translations generated from the bilingual corpus into the discriminator. The discriminator’s outputs are then used to compute the cross-entropy loss with $1$, which is fed back to the generator. The generator combines this loss with the translation task loss using a smaller weight. This approach aims to assist in improving the translation model. In this way, the generator continuously produces more natural sentences to deceive the discriminator.

By using monolingual corpora in this manner, we observe that synthetic sentences derived from monolingual translation not only avoid interfering with the translation model but also enhance the discriminator’s capability along with real sentences. In turn, the discriminator helps improve the performance of the generator. Thus, we achieve the goal of using monolingual corpora on the source side to improve the model’s translation ability without negatively impacting the translation model.

To demonstrate the effectiveness of the low-resource sentences translated by our generator, we utilize data augmentation experiments for verification. However, in a standard data augmentation process, the synthetic sentences may vary in quality, and we cannot ensure that all sentences in the synthetic corpus are of high quality. Based on this, this paper proposes an effective high-quality filtering process, as illustrated in Figure 3.

We use the German-Upper Sorbian (de-hsb) parallel corpus from WMT20¹¹1https://www.statmt.org/wmt20/unsup_and_very_low_res/ as the original bilingual corpus. The training sets of the dataset consist of 60,000 parallel sentences, while both the validation and test sets contain 2,000 sentences each. We choose German monolingual corpus from WMT14²²2https://www.statmt.org/wmt14/training-monolingual-news-crawl/ as the original monolingual corpus. This German monolingual corpus consist of 1,879,765 German sentences. We apply filtering and do not utilize all of the data. Since we will try to understand what is the necessary and sufficient amount of additional data to attain the best performance. The statistics of the corpora used are shown in Table 1.

Given the outstanding performance of Transformers in NLP tasks, especially NMT, we choose a vanilla Transformer model as our generator. Its encoder and decoder each have 8 heads and 12 layers in total. We set the embedding dimension to 512 and the dropout rate to 0.15. We employe a warm-up strategy, after which the learning rate decays with the inverse square root schedule. On the discriminator side, we use a parallel combination of CNN and BiLSTM. The CNN had a convolutional kernel size of $16\times 1$, while the BiLSTM had a hidden layer size of 256. Both the discriminator and generator use an Adam optimizer.

All of our models and training processes were implemented using PyTorch [11]. The FAISS toolkit [8] was utilized for similar sentence retrieval, while the kenLM toolkit [6] was employed for training $N$-gram models. All model training was conducted on a single A4500 GPU. To evaluate the translation results, we used BLEU [10], chrF2 [12], and TER [17] as the evaluation metrics. The calculation of these metrics were implemented through the sacreBLEU toolkit [13].

The first group of rows in Table 2 shows the performance of the generator with different components. In this series of experiments, we used a vanilla Transformer model as the baseline. This model, serving as the generator, achieved a BLEU score of 37.2 for translation. Building upon this baseline, we incorporated different components.

We first integrated TM into the input of the model. The results surpassed the baseline by 1.4 BLEU points. Considering the confidence intervals. This improvement is not significant, as the intervals for the baseline model (37.2 ± 1.2) and the model with TM integrated (38.6 ± 1.2) overlap (38.2 - 1.2 = 37.0 $<$ 38.4 = 37.2 + 1.2).

Subsequently, we trained the generator using a monolingual corpus in conjunction with a GAN. However, we found that the performance improvement was not significant (37.3 - 1.1 = 36.2 $<$ 38.4 = 37.2 + 1.2). We believe that this is due to the instability of GAN training, which also confirms that GANs are not well-suited for NMT tasks [25]. Of course, this is also related to the weight proportions of different loss values during the training process. Nevertheless, utilizing GANs was not our primary purpose. Our most important goal in employing GANs was to introduce source-side monolingual corpora into the model training process.

From the metrics, we can observe that when both TM and GAN are integrated into the system, the translation performance improves by 3.7 BLEU points compared to the baseline. This improvement is significant, as the intervals for the baseline model (37.2 ± 1.2) and the model with TM and GAN integrated (40.9 ± 1.2) do not overlap (40.9 - 1.2 = 39.7 $>$ 38.4 = 37.2 + 1.2). These four sets of experiments demonstrate that TM is an effective method for enhancing the translation performance of the model. We believe that this is because TM increases the amount of available data for model training and serves as a prompt for the model. The GAN structure, on the other hand, enables the utilization of source-side monolingual corpora. The synthetic sentences not only do not interfer with the generator but also indirectly facilitate the learning process of the generator through the discriminator.

For comparison with the method proposed by [15], a reverse translation model (from target to source) was trained in this study (see the 1st row in Table 2).

As shown in the second group of rows of the Table 2, we used the best performing generator from the first group of rows (combining TM and GAN) for translation and conducted data augmentation experiments following the process illustrated in Figure 3.

Initially, we augmented the original bilingual corpus with the entire monolingual corpus (60,000 German sentences) and their translations without any filtering (see the 6th row in Table 2). We found that the results hardly improved. After analysis, we concluded that this was due to the gap between the synthetic translation sentences and real natural language. Augmenting with an equal number of synthetic sentences as real sentences would interfere with the translation model, which is consistent with our previous analysis.

Therefore, we reduced the quantity and randomly selected 10,000 synthetic parallel sentence pairs for augmentation (see the 8th row in Table 2). As expected, the results showed preliminary improvement, surpassing the baseline by 1.0 BLEU point.

We then changed our filtering strategy to ensure the high quality for the selected sentence pairs. As introduced in Section 2.3, we employed three filtering methods using the sentence length ratio, sentence perplexity ratio, and a combination of length ratio and perplexity ratio from the original bilingual corpus as the filtering criterion. The experimental results demonstrate that using both length and perplexity ratios as the filtering criterion yields the best performance, surpassing the baseline by 2.9 BLEU points (see row number 2 and 13 in Table 2: $40.1-1.3=38.8>38.4=37.2+1.2$).

Additionally, to prove that the experimental results are not entirely influenced by the number of sentences, we randomly selected the same number of sentences (13,464) as the best-performing method (see the 11th row in Table 2).

For Sennrich’s method, 13,464 sentences were also randomly selected. In comparison, our method demonstrated better performance (see row number 12 and 13 in Table 2). However, the improvement was not significant. In the back-translation method employed by Sennrich, the target sentences are natural. Therefore, it can be concluded that after our high-quality filtering process, the synthesized target sentences also more closely resemble natural sentences. The results were lower than those obtained using the two filtering methods. Therefore, we conclude that our filtering process is effective.