Volctrans Parallel Corpus Filtering System for WMT 2020

We trained the word alignment model on the provided clean parallel corpus by using the fast-align toolkit (Dyer et al., 2013), and get the forward and reverse word translation probability tables. It’s worth mentioning that both of Pashto and Khmer corpus are tokenized before word alignment model training for accuracy consideration. We separate Pashto words by moses tokenizer¹¹1https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/tokenizer.perl. For Khmer, we use the character (\u200B in Unicode) as separator when it’s available and otherwise use a dictionary-based tokenizer by maximizing the word sequence probability.

This step is operated by our mining module. With the bilingual word translation probability tables, the mining module evaluates the translation quality of bilingual sentence pairs by YiSi-2 (Lo, 2019), which involves both lexical weight and lexical similarity. The Document pairs are first segmented on each language side using Polyglot ²²2https://github.com/aboSamoor/polyglot. This initial segmentation is represented as:

	$\displaystyle e$	$\displaystyle=\overline{e}_{1}\overline{e}_{2}\cdots\overline{e}_{a}=\overline{e}_{1}^{a}$		(1)
	$\displaystyle f$	$\displaystyle=\overline{f}_{1}\overline{f}_{2}\cdots\overline{f}_{b}=\overline{f}_{1}^{b}$		(2)

where $\overline{e}_{k}$ ($\overline{f}_{k}$) is a segment of consecutive words of document $e$ ($f$). Then we compute the sentence similarity (translation quality) by iteration from the initial segment ($\overline{e}_{1}$, $\overline{f}_{1}$). If the similarity reaches the preset threshold for ($\overline{e}_{i}$, $\overline{f}_{j}$), we pick the segment pair as parallel sentence candidate, and continue the computation from ($\overline{e}_{i+1}$, $\overline{f}_{j+1}$).

We notice that the inconsistency of segmentation in the document pairs can lead to the results: a sentence in one language contains information only part of a sentence in the other language, or two sentences (in different languages) both contain part of their information in common. These resulting sentence pairs may have low similarity scores.

In order to alleviate this problem, we also incorporate a parallel segmentation method in our mining module. We follow the basic idea proposed in (Nevado et al., 2003) where the parallel segmentation finding problem is treated as an optimization problem and a dynamic programming scheme is used to search for the best segmentation. We then briefly introduce our method. ³³3More details can be found in (Nevado et al., 2003)

After obtaining the monolingual initial segmentation $(\overline{e}_{1}^{a},\overline{f}_{1}^{b})$, a parallel segmentation is represented as:

$$\begin{split}s\equiv([\overline{e}_{1}^{k_{1}},\overline{f}_{1}^{j_{1}}],[\overline{e}_{k_{1}+1}^{k_{2}},\overline{f}_{j_{1}+1}^{j_{2}}],\\ \cdots,[\overline{e}_{k_{|s|-1}+1}^{k_{|}s|},\overline{f}_{j_{|s|-1}+1}^{j_{|}s|}])\end{split}$$

(3)

where $|s|$ is the number of segments for the parallel segmentation $s$ and $\overline{e}_{i_{1}}^{i_{2}}$ ($\overline{f}_{i_{1}}^{i_{2}}$) are consecutive segments $\overline{e}_{i_{1}}\overline{e}_{i_{1}+1}\cdots\overline{e}_{i_{2}}$ ($\overline{f}_{i_{1}}\overline{f}_{i_{1}+1}\cdots\overline{f}_{i_{2}}$). In this setting, all initial segments will be included in the parallel segmentation, and the order of the initial segments cannot be inverse. Therefore, the alignment is monotone.

Then we search for the best parallel segmentation using the objective function:

$$\max\limits_{S\in s}C\cdot\prod\limits_{n=1}^{|s|}P(\overline{f}_{j_{n-1}+1}^{j_{n}}|\overline{e}_{k_{n-1}+1}^{k_{n}})$$

(4)

where $P(\overline{f}_{j_{n-1}+1}^{j_{n}}|\overline{e}_{k_{n-1}+1}^{k_{n}})$ is the translation quality of the pair $(\overline{f}_{j_{n-1}+1}^{j_{n}},\overline{e}_{k_{n-1}+1}^{k_{n}})$.

Next, we use a dynamic programming algorithm to compute the best segmentation $S$ where we have a restriction that no more than 3 initial segments can be joined.

Finally, this set of parallel segments are combined with the set of the extracted initial segment pairs through global deduplication to serve as the output of our mining module.

It is worth noting that the sequence can be very long in the process of translation quality computation because several segments can be joined together. Therefore, while computing the similarity of a segment pair, our method based on word translation probability tables can be more time-efficient than LASER, as LASER is based on sentence embeddings and can be very slow when its LSTM encoder is fed with long sequence. Thus we do not consider using LASER to compute translation quality in our mining procedure.

The quantity and quality of the mined data are basically dependent on the word alignment model. Besides, more high-quality parallel corpus is used, the word alignment model would be more accurate and robust. Therefore, we propose an iterative mining strategy. For the first time, all the provided parallel data by the task are used to train the word alignment model. But we keep mining data for several times. We iteratively add new high-quality sentence pairs to the parallel corpus and train the word alignment model again to improve the word translation probability tables, thus boosting the mining cycle.

Recently, pre-trained transformer-based models play an important role in a variety of NLP tasks, such as question answering, relation extraction, etc.. Pre-trained models are often trained from scratch with self-supervised objective, and then fine-tuned to adapt to the downstream tasks. In our system, we choose the XLM (Conneau and Lample, 2019) as our main model. The reason are as follows: a) Similar to BERT (Devlin et al., 2019; Yang et al., 2019), XLM has Masked Language Model (MLM) objective, which enables us to make the most use of the provided monolingual corpora; b) XLM also has Translation Language Model (TLM) objective. Taking two parallel sentences as input, it predicts the randomly masked tokens. In this way, cross-lingual features can be captured; c) With a large amount of training corpus in different languages, XLM can provide powerful cross-lingual representation for downstream tasks, which is very suitable for parallel corpus filtering situations.

We follow the instructions ⁴⁴4https://github.com/facebookresearch/XLM to prepare the training data and train the XLM model. In detail, we use Moses tokenizer to tokenize the text ⁵⁵5We do not use the character-/dictionary- based method introduced in Section 2.2.1 to tokenize Khmer here. Performance may be improved with that method, but we have run out of time, unfortunately., and fastbpe ⁶⁶6https://github.com/glample/fastBPE to learn and apply Byte-Pair Encoding. We use $50$K BPE codes on the concatenation of all the training data. After applying BPE codes to the training data, we obtain a large vocabulary containing around $100$K tokens. Therefore, we only keep the top frequent $100,000$ tokens to form the vocabulary and train the XLM.

We use monolingual data in MLM objective and parallel data in TLM objective. In detail, the monolingual data we use are as follows:

• •

  Khmer: all the $14$M provided sentences.

• •

  Pashto: all the $6.6$M provided sentences.

• •

  English: because the number of english monolingual sentences are so large, we subsample $25$M sentences to keep a balance.

All the available parallel sentence pairs ($29$K en-km and $12$K en-ps) are used in TLM objective. For each objective, we hold out $5$K sentences or sentence pairs for validation and $5$K for the test.

We pre-train the XLM using two different settings on $8$ Tesla-V100 GPU: a) Standard: The embedding size is $1024$, with $12$ layers and $64$ batch size. b) Small: The embedding size is $512$, with $6$ layers and $32$ batch size. The other values of hyperparameters are all set to the default values. The two pre-trained XLM model is then fine-tuned in downstream task and further ensembled.

To score sentence pairs according to their parallelism, classification models are usually used (Xu and Koehn, 2017; Bernier-Colborne and Lo, 2019). In the training phrase, it is formulated as a binary classification problem, whether the sentence pair is semantically similar to each other or not. In the inference phrase, the probability of the positive class is considered as the score of the sentence pair. Therefore, we use the provided parallel sentence pairs as the positive instances, and construct negative instances taking advantage of such positive instances similar to Bernier-Colborne and Lo (2019). Specifically, we generate negative examples in the following ways:

• •

  Shuffle the sentences in source language and target language respectively, and randomly align them.

• •

  Randomly truncate the length of the source sentences or/and target sentences to $3$.

• •

  Randomly shuffle the order of the source sentences or/and target sentences.

• •

  Simply swap the source and target sentences. Or replace the source/target sentences with target/source sentences, such that the two sentences are exactly the same.

We only add a linear or convolutional layer on top of the pre-trained XLM model and predict through a sigmoid function. The input of the model is the concatenation of a sentence pair, separated by one [SEP] token. Besides, to tackle the problem that some sentences may be too long, we simply truncate each sentence such that the maximum length of sentence is $128$. The dropout rate is set to $0.5$.

We apply some reranking mechanisms in order to compensate for the latent bias in the XLM-based scorer, and aim to boost the quality of the whole corpus rather than each sentence pair independently.

The first reranking mechanism is based on language identification. For some sentences, they may include many tokens that do not belong to the corresponding language, and therefore damage the performance of the machine translation system. This phenomenon is rather common in Khmer-English corpus in particular. We utilize pycld2 tools ⁷⁷7https://github.com/aboSamoor/pycld2 to identify the language of the sentences. The scores of those which cannot be identified as the corresponding language are reranked by a discount of $\alpha$. $\alpha$ is a hyperparameter.

The second reranking mechanism is based on n-gram coverage. Because the sentence pairs are scored independently, redundancy may exist in those high-score sentences. To enhance the diversity of the selected corpus, we first sort the sentence pairs in the descending order based on their scores. Next we maintain a $n$-gram pool for source sentences, and scan the source sentences from the top down. Those sentences that have no $n$-gram different from those in the pool will receive a discount of $\beta$, and both $n$ and $\beta$ are hyperparameters.

Note that before reranking, we always normalize the score according to their rankings, so that scores provided by different models can be unified. The score of the $i$-th sentence pair is:

$$score_{i}=1-\frac{rank_{i}}{N}$$

(5)

where $i$-th pair ranks $rank_{i}$ in all the sentence pairs and $N$ denotes total number of pairs.

We also try to rerank through language models, but it does not bring improvements. Thus we do not use this reranking mechanism in our submissions.

Different models may capture different features during training and inference. To make use of group wisdom and improve the final performance, we ensemble the following four models by averaging scores:

• •

  Model 1: Standard XLM + Linear Layer. The learning rate of XLM and linear layer are $1e^{-8}$ and $1e^{-5}$ respectively.

• •

  Model 2: Standard XLM + Linear Layer. The learning rate of XLM and linear layer are $1e^{-7}$ and $1e^{-4}$ respectively.

• •

  Model 3: Standard XLM + Convolutional Layer. The learning rate of XLM and linear layer are $5e^{-7}$ and $5e^{-4}$ respectively.

• •

  Model 4: Small XLM + Linear Layer. The learning rate of XLM and linear layer are $5e^{-7}$ and $5e^{-4}$ respectively.

All the models use $16$ batch size per GPU.