Revisiting Round-Trip Translation for Quality Estimation

One goal of QE is to estimate the quality for machine translated sentences without reference translations, but the definition of quality has gradually changed. Traditional QE aimed to estimate the required amount of post-editing efforts for a given translation in the word, sentence or document level. In the sentence level, this can be understood as estimating the Human Translation Error Rate (HTER) [Snover et al., 2006], or the rate of edit operations which include the insertions of words, deletions or replacements. The recently proposed view of "QE-as-a-metric" [Fonseca et al., 2019]¹¹1Since WMT20, this was modified to the ”sentence-level direct assessment task”. differs from traditional quality estimation in that it directly aims to estimate the absolute score of a translation, and can be directly compared with previous reference-based metrics. While reference-based metrics easily achieve above 0.9 Pearson correlation with direct human assessments in the system-level and up to 0.4 correlation in the sentence-level, QE-based metrics typically score less [Ma et al., 2019].

YiSi [Lo, 2019] is the best performing QE metric from the recent QE-as-a-metrics subtask submitted to the quality estimation shared task of WMT19 [Ma et al., 2019]²²2We excluded UNI and its variants from consideration, since they do not have any open publications to refer to. See Table 2 in [Ma et al., 2019].. It takes contextual embeddings extracted from Bert and computes F-scores of semantic phrases using the cosine similarity of words weighting by their inverse document frequency (idf). YiSi has variants for both situations where the references exist (YiSi-1) or does not exist (YiSi-2).

RTT had frequently been used for a means of evaluating MT systems until Somers [Somers, 2005] and van Zaanen and Zwarts [van Zaanen and Zwarts, 2006] claimed that RTT is inappropriate as a QE metric for translations. The idea was supported by the low correlations between a Bleu score for the input and RTT (RTT-Bleu) and a Bleu score for the reference and output. However, Bleu is not an adequate metric to validate RTT for QE. When Aiken and Park [Aiken and Park, 2010] re-assessed RTT with human judgments, there was a significant positive correlation between the human scores of round-trip translations and one-way translations.

Recently, RTT has been employed for other purposes: generating paraphrased sentences and modeling purposes. Yu et al. [Yu et al., 2010] exploit RTT-based features to estimate the quality of spoken language translation and improve the accuracy of QE model. Mallinson et al. [Mallinson et al., 2017] reassess using RTT for generating paraphrases in the context of NMT. Junczys-Dowmunt and Grundkiewicz [Junczys-Dowmunt and Grundkiewicz, 2017] and Lichtarge et al. [Lichtarge et al., 2019] generate large amounts of artificial data to train an automatic post editing model and grammatical error correction, respectively. Vaibhav et al. [Vaibhav et al., 2019] also uses RTT to augment bilingual data for NMT. Lample et al. [Lample et al., 2018] measures RTT-Bleu for model selection purposes and Hassan et al. [Hassan et al., 2018] uses RTT as a feature to re-rank translation hypotheses.

Lexical metrics, such as Bleu [Papineni et al., 2002] and chrF [Popović, 2015], have long and widely been used for translation evaluation. Both metrics compute strict matching between translation output and reference at the surface level. Bleu counts the n-gram matches of the output and reference over the number of tokens of output as well as the length similarity of the output and reference. chrF computes F-score based on character-level n-grams. However, they cannot capture the semantic similarity of output and reference sentences beyond lexical relatedness or overlap. In this sense, lexical-based metrics may not be the best way to measure the similarity of paraphrases.

Bert [Devlin et al., 2019], a pre-trained language representation model, made breakthroughs on many natural language processing tasks, including the sentence similarity prediction task [Cer et al., 2017]. The methods using Bert’s embedding vectors were also introduced to MT evaluation, the task that needs semantic-level similarity measurement, and show the best performance [Ma et al., 2019, Lo, 2019, Zhang et al., 2019a]. BERTScore [Zhang et al., 2019a] leverages Bert wordpiece embeddings to compute sentence similarity of two monolingual sentences. When BERTScore is applied to the output and reference, it outperforms Bleu and chrF. Meanwhile, sentence-Bert (SBert) [Reimers and Gurevych, 2019], a fine-tuned Bert, is introduced to derive more semantically meaningful sentence-level representation than Bert. From the encouraging results of the embedding-based methods, we would expect the embeddings to catch the semantic similarity of input and round-trip sentences.

In our settings, the semantic-level metrics are represented by the cosine similarity of SBert embeddings and BERTScore. For all metrics, system-level score is an averaged sentence-level scores.

To estimate the quality of MT systems with RTT, a BT system is required. The choice of the BT system seemingly has the potential to largely affect the performance of RTT-based QE metrics, so we run experiments to verify the effect of BT system qualities. We compare two types of models–the system trained solely on WMT19 news translation task training corpus and online system–with different performance in terms of Bleu. The BT systems trained on the WMT19 news dataset could be considered adequate to evaluate the WMT19 submitted FT systems since both systems are trained on the same domain. On the other hand, online systems could also be desirable, because the online systems are trained on a huge amount of corpus mixed with various domains and would outperform the trained models on WMT19 dataset. If the online systems show more favorable results, then RTT-based QE metrics can be more practical in terms of the easy access to a BT system on any language pair.

To examine the impact of the BT systems, we choose English–German, which is the most submitted language pair. For trained BT systems, we use Facebook-FAIR⁵⁵5Submitted model is publicly available via PyTorch
(https://pytorch.org/hub/pytorch_fairseq_translation)., the best system in WMT19 on German–English, and the Transformer Big model [Vaswani et al., 2017] saved at 10k, 20k, 40k, and 100k iterations during training on the WMT19 corpus. Details of the Transformers are described in Appendix B. We also try three online systems, namely Google, Microsoft, and Amazon, showing different Bleu on WMT19 German–English evaluation set. Each system was requested on Oct 2019, Nov 2019, and Dec 2019.

The metric might penalize or favor a certain type of models. For instance, Bleu has been argued to penalize rule-based systems against statistical systems [Hovy, 2007].

To investigate whether RTT-based QE metrics penalize FT systems based on their architecture, we assess RTT-based QE metrics on both NMT and SMT. As the all models submitted to WMT19 are NMT [Ma et al., 2019], and the models submitted to WMT12 are SMT or rule-based model [Callison-Burch et al., 2012], we denote the former as NMT and the latter as SMT. We compare RTT-based QE metrics’ performance with Pearson correlation coefficient for the language pairs both appeared on WMT19 and WMT12, English–Czech, English–German, and German–English.

Due to the nature of RTT-based QE metrics, a BT system is needed. We use a variety of BT systems in terms of the training recipe and Bleu on WMT19 German–English testset and observe the performance of RTT-based QE metrics evaluated by Pearson correlation ($r$) with human scores [Ma et al., 2019, Ma et al., 2018]. Note that well-performing metrics achieve high correlation coefficient.

According to Table 2, RTT-BERTScore and RTT-SBert not only outperform the other metrics but are also robust to the type and performance of the BT systems. On the other hand, RTT-Bleu and RTT-chrF are sensitive to the performance of the BT systems, and the correlations fall behind RTT-BERTScore and RTT-SBert. Since BT systems scoring low Bleu have less chance of having same word orders in RTT as with input sentences, the performance of surface-form metrics, RTT-Bleu and RTT-chrF, decrease more sharply than RTT-SBert and RTT-BERTScore.

The best correlation of each metric is accomplished when the online system is used for the BT system. Even though Microsoft and Facebook-FAIR exhibit a similar Bleu score, metrics are more successful when using the Microsoft system. This can be explained by a variance of RTT-based QE metrics score. In average, the variance of RTT-SBert and RTT-BERTScore using the online BT systems is higher than that of trained ones. The trained BT systems might over-translate a fault translation output similar to the original input, e.g., Kim Jong Un – Kim – Kim Jong Un, that make QE metrics hard to distinguish good systems to the bad ones.

Surprisingly, the best BT system in terms of Bleu does not always guarantee the best RTT-based QE metrics. Despite Google’s highest Bleu score, the performance of RTT-based QE metrics is lower than or similar to that of Microsoft. This assures that Bleu is not the only feature that affect the performance of the RTT-based QE metrics.

Provided from the results in Section 5.1, we use one of the online systems to get RTT for all language pairs in WMT19. Specifically, we use Google Translate, because of its coverage of supported language pairs and its overall performance across all language pairs.

We conduct the same experiments as in the WMT19 metrics shared task to directly compare with the previous QE-as-a-metrics. Individual best results of previous methods and YiSi-2 are provided to compare RTT-based QE metrics within the same reference-free metrics. Note that YiSi was the only QE-as-a-metric scoring on all language pairs, at the same time, achieving the best performance in total [Ma et al., 2019]. We also include commonly used reference requiring metrics, namely Bleu, chrF, sacreBleu-Bleu, and sacreBleu-chrF, to see how far QE metrics can get without reference translation. The metrics are evaluated in system-level and sentence-level for all language pairs. Specifically, Pearson correlation is applied to assess system-level metrics and Kendall’s $\tau$-like formulation against the daRR to measure sentence-level metrics.

Table 3 illustrates the system-level correlations with human judgments on both to-English and out-of-English language pairs. Across all language pairs except German–English, Bert-based RTT QE metrics outperform RTT-Bleu and RTT-chrF. For some language pairs, Gujarati–English, Kazakh–English, Russian–English, English–German, English–Gujarati, and English–Kazakh, RTT-Bert-based metrics show comparable result to Bleu, however, QE metrics still fall behind the reference-based metrics on average. Surprisingly enough, the high Pearson correlation coefficients are mostly achieved on low-resource language pairs. Results in Figure 1 suggest that this might due to the high variance of the system’s DA scores which implies distinguishing good systems to the bad ones is relatively easy.

To present a more reliable view, we draw plots of Pearson correlation while reducing MT systems to top $n$ ones as in Ma et al. [Ma et al., 2019]. Figure 2 depicts English–German, and all language pairs are in Appendix C. In general, correlations of sacreBLEU-BLEU and RTT-based QE metrics tend towards 0 or negative, whereas the reference-based metric shows a rather continuous degradation than RTT-based metrics. RTT-SBert and RTT-BERTScore are better at retaining positive correlations compared to RTT-Bleu and RTT-chrF, however, their consistency is weaker than sacreBLEU-BLEU except for some language pairs (English–German, English–Gujarati, English–Kazakh, and Finnish–English).

Metrics performance on sentence-level is described in Table 4⁶⁶6Instead of Bleu, we report sentBleu.. Sentence-level quality estimation is considered as a more difficult task than that of system-level. This is supported by the poor correlation coefficients of even sentBleu and chrF. Similar to the system-level results, QE metrics fall short of the reference-based metrics. For language pairs with high DA score variance, again, RTT-Bert-based metrics provide comparable performance with reference-based metrics.

A certain type of FT system could be penalized by one metric according to its computation method. For this reason, we observe the performance of RTT-based QE metrics on different FT systems: SMT and NMT. SMT denotes the systems submitted to WMT12 and NMT represents the systems submitted to WMT19. Same as the Section 5.2, we use Google Translate for the BT system and evaluate the metrics with Pearson correlation coefficient. Results are shown in Table 5.

RTT-SBert and RTT-BERTScore demonstrate the most promising performance regardless of the FT systems. In contrast, RTT-Bleu and RTT-chrF seem to favor SMT. The correlation coefficient gap between Bleu and both RTT-Bleu and RTT-chrF is smaller when FT system is SMT.

The results from all the previous sections consistently show the outstanding performance of RTT-SBert and RTT-BERTScore. We see this in a view of paraphrase detection ability of SBert and BERTScore. To confirm our assumption, we compare the area-under-curve (AUC) scores of precision-recall curves of the four metrics used to measure input and RTT on PAWS dataset. The higher the score is, the better the metric at paraphrase detection. Table 8 depicts the results. Note that SBert indicates the cosine similarity of the embedding vectors of two sentence pairs extracted from the model.

As expected, BERTScore and SBert outperform sentBLEU and chrF. In case (a) of Table 6 and case (d) of Table 7, we can find sentBLEU and chrF are sensitive to the change of word order. Additionally, they are hard to distinguish paraphrases on long sentences. From case (b), (c), (d), and (e), lexical-based metrics constantly view the sentences are not paraphrased.

The results imply that metrics capability to measure the semantic similarity is highly correlated to the performance of RTT-based QE metrics.