VBD-MT Chinese$\leftrightarrow$Vietnamese Translation Systems for VLSP 2022

For bilingual data, there are 300K (thousands) parallel sentences provided by the shared task. This can be seen as a low-resource machine translation task in comparison with large-scale bilingual training data, which contain millions of pallel sentences such as Czech-English, English-Chinese, French-German  Barrault et al. (2020); Akhbardeh et al. (2021). The limited data causes the task challenging but interesting to investigate appropriate approaches and models.

In addition to the bilingual data, the shared task also provided large scale monolingual datasets including 19M (millions) sentences for Chinese and 25M sentences for Vietnamese, which can be utilized as an additional resource to enhance translation models.

The monolingual datasets are used for backtranslation (Section 3.3). By analyzing training and validation datasets, we found that most sentences are in the length of 10 to 60 words (cover more than 90%). Therefore, we first quickly apply a filtering method by sentence length, where keeping only sentences in the sentence length range of 10-60. By doing this, we obtained 6M sentences. Due to time limitation, we do not make use the whole but a subset of the data. We randomly selected 1.5M sentences from the filtered 6M data for backtranslation.

For evaluation, the validation set and public test sets contain 1K sentences. We used the officially provided validation set for tuning hyper-parameters and saving model checkpoints, and the systems are compared based on the public test sets.

We present the statistics of these datasets in Table 1. ²²2The vocabulary size is an estimation, when we calculate each Chinese character as a token, when Vietnamese tokens are split by white spaces.

We build our baseline systems based on the well-known and powerful Transformer model Vaswani et al. (2017). Our systems are based on the Fairseq Ott et al. (2019) PyTorch implementation, which has achieved state-of-the-art (SOTA) performance on competitions Tran et al. (2021); Akhbardeh et al. (2021). This is confirmed by our preliminary experiments when we compared several systems to select a strong baseline (we present the detail results in Section 4.1). The model is optimized using the Adam optimizer Kingma and Ba (2014). We used learning rate $3e-5$, dropout $0.3$, trained with early stopping, and saved models after every $5,000$ steps. We trained our models on 1 NVIDIA GeForce 11GB GPU. We present detail hyper-parameters in Appendix A.

The model is finetuned on the mBART Liu et al. (2020), which is trained on 25 languages (mBART-25)³³3We also tried the mBART-50 Tang et al. (2020), which is trained on 50 languages, but our preliminary experiments showed that the mBART-25 is better. including Chinese and Vietnamese.

We tokenized texts by the well-known SentencePiece Sennrich et al. (2016b). For evaluation metric, we used the commonly used SacreBLEU Post (2018).

We used the mBART-25 Liu et al. (2020) for finetuning the baseline systems. The vocabulary size is 250K, which caused the out-of-memory issue when we trained on our NVIDIA GeForce 11GB GPU. We therefore filtered the vocabulary, which contains only the vocabulary of the provided bilingual datasets and the 6M filtered monolingual datasets. Only the embeddings correspond to the filtered vocabulary are kept. As a result, the filtered vocabulary size becomes 67K, which is approximately four times smaller than the original vocabulary, and we can train our systems without a more powerful server. This technique is a minor step, but it may be useful for training systems on limited computational resources.

Leveraging available large scale monolingual data via backtranslation to improve MT systems is a common strategy and has been utilized in previous work Sennrich et al. (2016a); Tran et al. (2021); Akhbardeh et al. (2021). In this work, we utilize backtranslation using the top-$k$ sampling method Edunov et al. (2018) to generate synthetic data, which select the top-$k$ highest scoring outputs ($k=5$) at every time step. The method has shown to provide richer training signal, which is better than the maximum a posteriori (MAP) beam search, which reduces the diversity and richness of the generated source translations.

We used the baseline systems trained on the provided bilingual data to translate the selected 1.5M monolingual data. As a result, we obtained 211K (zh-vi) and 403K (vi-zh) sentence pairs for back-translated datasets. Table 2 describes the back-translated data.

Ensembling weights is also a strategy to improve machine learning models Shahhosseini et al. (2022) including MT systems Tran et al. (2021). In our systems, we save different epoch checkpoints and calculate the ensembled model by calculating the average of models’ weights of the last $N$ checkpoints. We tried different values of $N$ and found that $N=5$ is the best value for our systems.

Translating numbers look straightforward but has been shown as a challenging task Wang et al. (2021). When we conducted analyses on translation output, we found that specific data types such as date-time and numeric values can be mis-translated, which is a big problem when translating the wrong values of human or currency. The task can be solved by combining with named entity task Mota et al. (2022), but we leave such method for another extended work. In the scope of this work, we simply create a set of patterns for post-processing to edit the translations of date-time and numeric values. For instance, we illustrate several patterns such as the followings.

• •

  XY亿 $\rightarrow$ XY $*100$ millions

• •

  XY万 $\rightarrow$ XY $*10$ thousands

The patterns are simple yet effective. We discuss the results with corrected samples in Section 4.3.

In order to investigate and build a strong baseline for this task, we first review and compare several existing and recent proposed models. In particular, we compared the following systems.

• •

  Fairseq Ott et al. (2019): a PyTorch implementation, which has shown the SOTA performance on competitions including the well-known WMT shared tasks Barrault et al. (2020); Akhbardeh et al. (2021).

• •

  RAML Norouzi et al. (2016): used a reward augmented maximum likelihood, which tries to optimize task reward (loss) used for test evaluation, and shown the contribution for neural sequence to sequence models including machine translation.

• •

  Point.Gen Enarvi et al. (2020): uses a pointer-generator network to facilitate the same parts of a source sentence (such as person names) to a target sentence.

• •

  Bi-SimcutGao et al. (2022): a training strategy using a regularization method to forces the consistency between the output distributions of the original and the cutoff sentence pairs to boost translation performance.

• •

  Multi-Stage Prompting (MSP) Tan et al. (2022): a method uses different continuous prompts for shifting from pre-trained models to translation tasks better.

It is noted that for RAML, Bi-Simcut, and Pointer Generator, we used the Huggingface’s implementation.⁴⁴4https://github.com/huggingface/transformers

The compared results are presented in Table 3. From these preliminary experiments, we found that the Fairseq obtained the SOTA performance on the public test set. Therefore, we selected the Fairseq implementation for our baseline system.

We conducted a post-processing step to correct the translation output, which we focus on correcting numeric and date-time values. We manually check the output of the test set and found that mis-translated output can be can be edited correctly. For instance, the number 400亿 (40 billion) and date 2021年12月1日 (December 1st, 2021) (as presented in Table 6) are correctly revised via the post-processing step.

Besides the systems’ BLEU performance, we would like to analyze the translation quality from the expert viewpoint. Therefore, we conducted human evaluation to investigate the actual translation output. It is noted that this is not human evaluation conducted by the shared task organizers, but we invited a native speaker, who is fluent in both Chinese and Vietnamese to check and analyze a small set of the translated sentences.

In particular, for each translation task, we randomly selected a small set (20 translated samples) for the human expert to evaluate and analyze. For each sample, there are two outputs, which are from the baseline and the submitted systems, and we shuffled the outputs so that the expert does not know an output comes from which system. We let the expert to assign a score for each output (scoring $1-10$ point), and give comments for each output. From this evaluation, we gain several following observations.

• •

  The submitted systems produce better translations than the baseline systems (received higher scores) in most cases (more than $70\%$ of the samples), which confirm the improvement shown in BLEU scores.

• •

  The meaning of current translated output is acceptable to some extent when most cases receive the scores from $7$ to $10$ ($80\%$ of the samples).

• •

  Common issues in translations still remain such as: incorrect translations of person names; only correctly translate a part of content or missing main content, unclear translations; using inappropriate translated words, etc.

Though we achieve promising results with the baseline systems and improvement from our approaches, there are still several limitations. First, experiments on comparing systems to choose a strong baseline are yet fully completed. Second, we only leveraged a subset of the monolingual data (about 10% or less of the provided data), which has not completely utilized the large scale available data. Third, the performance may be better if we use the full vocabulary of the pretrained mBART model instead of pruning the vocabulary, although we are able to get benefits from this pruning when powerful servers are unavailable. Fourth, though we achieve several interesting and useful analyses from the human evaluation, it is better to conduct the evaluation on a larger number of samples.