Unsupervised Parallel Corpus Mining on Web Data

In this section, we will test the proposed pipeline on three language pairs, English-French, English-German, and English-Romanian. In the first step to generate pseudo parallel corpus ${\mathcal{A}}_{(p,q)}$, we follow the training script in the XLM repository⁴⁴4https://github.com/facebookresearch/XLM to train the unsupervised machine translator. Next, we sample 1M sentence from NewCrawl⁵⁵5https://www.statmt.org/wmt16/translation-task.html datasets of French, German and Romanian, and translate them into English by the unsupervised machine translator ${\mathcal{F}}$ to obtain ${\mathcal{A}}_{(p,q)}$.

For the URL domains feed into the crawler, we follow the ones used in the ParaCrawl project Esplà-Gomis et al. (2019), whose statistic information is included in the table 1 .

For the finetuned XLM model in the filtration step, we use the pretrained 6-layer XLMs, which are the same ones in the first step, as the initial parameters, then finetune them on the ${\mathcal{A}}_{(p,q)}$ for 10 epochs. The hyperparameter setting is the same as the XNLI finetuning script in the XLM repository.

In the table 2, we summarize the result of unsupervised web parallel data mining pipeline. Firstly, we observe that the size of crawled data has a similar scale of supervision data in WMT benchmark. Here, the WMT of EN-Fr indicates WMT2014 training set, and WMT of En-De and En-Ro are WMT2016 training set. Secondly, The result of the filtration process, comparing the size of ${\mathcal{B}}_{(p,q)}$ and ${\mathcal{C}}_{(p,q)}$, indicates 40%-50% crawled data are not high-quality parallel data.

In the following parts, we are going to evaluate the quality of this parallel corpus ${\mathcal{C}}_{(p,q)}$ by using it to train neural machine learning systems and compare the system performance on the supervised and unsupervised machine translation benchmark results.

Firstly, we evaluate the parallel corpus ${\mathcal{C}}_{(p,q)}$ with the supervised machine translation benchmark. We follow the experiment setting in the Scaling NMT paper Ott et al. (2018), including model architecture and choice of the hyper-parameters, and report the BELU score on the En-Fr and En-De directions on the WMT2014 test sets.

The evaluation results are included in table 3. WMT indicates that the model trained with the WMT training set. bt means the back-translation augmentation. From the results, we obverse that the machine translation system trained with ${\mathcal{C}}_{(p,q)}$ can achieve similar performance to the ones trained with millions of human-labeled parallel samples. The performance gap is small than 1 BELU score It indicates that the quality of ${\mathcal{C}}_{(p,q)}$ is similar to the current largest-scale public parallel dataset, while the proposed website data mining pipeline does not require any labeled parallel sample and dictionary as the seed.

Next, we evaluate our corpus on the benchmark setting of unsupervised machine translation problems. Similar to its problem definition, our pipeline can train a machine translation system without requiring any labeled parallel samples. The model architecture design and choice of the hyper-parameters are the same as XLM Lample and Conneau (2019) and MASS Song et al. (2019) papers. The machine translation systems are trained with ${\mathcal{C}}_{(p,q)}$ and the back-translation augmented data generated in an online manner. The (En, Fr) results are the BLEU scores on the WMT2014 test set. The (En, De) and (En, Ro) results are the BELU scores on the WMT2016 test set.

The experiment results are included in table 4. Compared with both baselines, the model trained with data from the proposed pipeline achieves a large margin improvement in all directions. The proposed method averagely improves 4.55 BELU scores compared with the best baseline. In the low resource case, Ro-En, our result, 38.95 BELU score, achieves new state-of-the-art results, even compared with the best performance with the WMT supervision data, which is 38.5 BELU score.

To better understand the importance of the crawler and filtration components, we perform an ablation study by eliminating the parallel data classifier in the filtration process from the proposed pipeline. We train three models respectively with the filtered parallel data ${\mathcal{C}}_{(p,q)}$, raw parallel data ${\mathcal{B}}_{(p,q)}$, and the low quality data ${\mathcal{B}}_{(p,q)}-{\mathcal{C}}_{(p,q)}$, which are the samples discarded by the classifier. The experiment setting is same as the supervised machine translation study in section 3.3. The experiment results are present in table 5. Surprisingly, trained with raw parallel data, the model can achieve similar performance compared to the filtered version, where the difference is smaller than 1 BELU score. On the other hand, the models trained with low-quality parallel data have significantly lower performance. It indicates that the filtration process can differentiate the quality of the parallel samples, but leaving this noise in the neural machine translator training process would not harm the final performance too much.