cushLEPOR: customising hLEPOR metric using Optuna for higher agreement with human judgments or pre-trained language model LaBSE

Machine Translation (MT) is a rapidly developing research field that plays an important role in NLP area. MT started from 1950s as one of the earliest artificial intelligence (AI) research topics and gained a large improvement in the output quality in large resourced language pairs after the introduction of Neural MT (NMT) in recent years Kalchbrenner and Blunsom (2013); Cho et al. (2014); Bahdanau et al. (2014). However, the challenge still remains in achieving human parity of MT output Han et al. (2021a). Thus MT evaluation (MTE) continues to play an important role in aiding MT development from the aspects of timely and high quality evaluations, as well as reflecting the translation errors that MT systems can take advantages of for further improvement Han et al. (2021b). On one hand, human evaluations have long been criticised as expensive and unrepeatable. Furthermore, the inter- and intra-agreement levels from Human raters may struggle to achieve a consistent and reliable score, unless done in rigour with highly trained and skilled evaluators Alekseeva et al. (2021). On the other hand, even though researchers have claimed that the automatic evaluation metrics have reached much better performances in the category of system level evaluations of MT outputs, with high correlation to human judgements, the segment level performance is still a large gap from human experts’ expectation Freitag et al. (2021); Barrault et al. (2019, 2020); Han et al. (2013a); Macháček and Bojar (2013).

In the meantime, many pre-trained language models have been proposed and developed in very recent years and showing big advantages in different NLP tasks, for instance, BERT Devlin et al. (2019) and its further developed variants Feng et al. (2020). In this work, we take the advantages of both high performing automatic metric and pre-trained language model, aiming at one step further towards higher quality performing automatic MT evaluation metric from both system level and segment level perspectives.

Among the evaluation metrics developed recent years, hLEPOR Han et al. (2013b, a) is an augmented metric that include many evaluation factors with tunable weights assigned including precision, recall, word order (via position difference factor), and sentence length. It has also been applied by researchers from different NLP fields including natural language generation (NLG) Novikova et al. (2017); Gehrmann et al. (2021), natural language understanding (NLU) Ruder et al. (2021), automatic text summarization (ATS) Bhandari et al. (2020), and searching Liu et al. (2021), in addition to MT evaluation Marzouk (2021).

However, original hLEPOR has disadvantage of the manual tuning of its parameter weights which take a lot of human efforts. We choose hLEPOR Han et al. (2013b, a) as our baseline model, and use the very recent language model LaBSE Feng et al. (2020) to achieve automatic tuning of its parameters thus aiming at reducing the evaluation cost and further boosting the performance. This system description paper is based on our earlier work, especially the training models Erofeev et al. (2021).

The rest of the paper is organised as below: Section 2 revisits hLEPOR metric, its factors, advantages and disadvantages, Section 3 introduces our Python ported version of hLEPOR and the further customised hLEPOR (cushLEPOR) using language models, Section 4 presents our experimental development and evaluation that we carried out on cushLEPOR metric using WMT historical data, Section 5 reserves space for our submission to this year WMT21 metrics task, and Section 6 finishes this paper with discussions of our findings and possible future work.

hLEPOR is a further developed variant of LEPOR Han et al. (2012) metric which was firstly proposed in 2013 including all evaluation factors from LEPOR but using harmonic mean for grouping factors to produce final calculation score Han et al. (2013b). Its submission to WMT2013 metrics task achieved system level highest average correlating scores to human judgement on English-to-other (French, Spanish, Russian, German, Czech) language pairs by Pearson correlation coefficient (0.854) Han (2014); Macháček and Bojar (2013). Other MT researchers also analysed LEPOR metric variant as one of the best performing segment level metric that was not significantly outperformed by other metrics using WMT shared task data Graham et al. (2015). hLEPOR is calculated by:

The basic version of hLEPOR carries out similarity calculation between MT system outputs and reference translations, in the same language setting, based on the word surface level tokens. The hybrid hLEPOR metric also carries out similarity calculation based on POS sequences from system-output and reference text. To do this, POS tagging is needed as the first step, then hLEPOR(POS) calculation uses the same algorithms used for the word level similarity score hLEPOR(word). Finally, hybrid hLEPOR is a combination of both word level and POS level score. In this system submission work, with the time limitations, to make an easier to use customised hLEPOR, we take the basic version of hLEPOR, i.e. the word level similarity calculation and leave the hybrid hLEPOR into the future work.

The weighting parameters for the three main factors in original hLEPOR metric, i.e the ($w_{LP}$, $w_{NPosPenal}$, $w_{HPR}$) set, in addition to the other parameters inside each factor, were tuned by manual work based on development data (see Appendix for detailed parameter value sets on each language pair for word-surface level evaluation, en $\Leftrightarrow$ cs/fr/de/es/ru). This is very time consuming, tedious, and costly. In this work, we will introduce an automated tuning model for hLEPOR to customise it regarding deployed language pairs, which we name as cushLEPOR.

The training and development data we used regarding MQM scores and pSQM labels is from the recent work by Google Research team on investigating into human evaluations based on WMT2020 shared task Freitag et al. (2021) (data available at https://github.com/google/wmt-mqm-human-evaluation).

We first focus on English-to-German language (EN-DE) pair, which includes MQM and pSQM labels, acquired from 10 submission of WMT 2020, then take ZH-EN data-set. We refer to the paper Freitag et al. (2021) for detailed MT system names and offering institutions.

Firstly, a multi-parameter optimisation against LaBSE for EN-DE language pair gave the following values for cushLEPOR parameters:

• •

  alpha = 2.97,

• •

  beta = 1.97,

• •

  n = 4,

• •

  weight_elp = 1.0,

• •

  weight_pos = 14.97,

• •

  weight_pr = 2.2.

This set of values reflected very different weighting systems in comparison to the original hLEPOR metric. For instance, 1) cushLEPOR assigned recall and precision much closer weight (2.97 vs 1.97) in comparison to hLEPOR (9.0 vs 1.0), 2) cushLEPOR chose 4-gram in chunk matching instead of bi-gram used in hLEPOR, 3) cushLEPOR assigned NPosPenal (n-gram position difference penalty) factor a very heavy weight against other two factors LP (length penalty) and HPR (harmonic mean of precision and recall) by (14.97 vs 1.0 and 2.2) in comparison to hLEPOR which emphasised the weight on HPR (1.0 vs 2.0 and 7.0). From these points of view, cushLEPOR trained on EN-DE language pair indicates the importance of the larger window context consideration during word matching, as well as the word order information reflected by n-gram (n value) and novel factor NPosPenal introduced by hLEPOR respectively.

This also reflected that LaBSE similarity is indeed a feasible goal for cushLEPOR optimisation. The correlations of hLEPOR and cushLEPOR to LaBSE are shown in Fig. 1 and 2.

However, we found out that we were not able to decrease much on RMSE (Root Mean Square Error) score for cushLEPOR towards pSQM , in comparison to original hLEPOR, (0.28 vs 0.29)which does indicate that original hLEPOR empirically shows very good fit for pSQM type human evaluation, using the suggested parameter settings for EN-DE Han et al. (2013a, b) as bellow.

• •

  alpha = 9.0,

• •

  beta = 1.0,

• •

  n = 2,

• •

  weight_elp = 3.0,

• •

  weight_pos = 7.0,

• •

  weight_pr = 1.0.

The RMSE value between pSQM and hLEPOR, vs pSQM and cushLEPOR is shown in Fig. 3. However, it indeed shows much better performance then BLEU metric, as in Fig. 4 (0.28 vs 0.46).

Optuna did optimise cushLEPOR against LaBSE very well, halving the RMSE distance between LaBSE and cushLEPOR as compared to original hLEPOR, shown in Fig. 5.

The performances of tuning on LaBSE and pSQM are shown in Fig. 6 and 7 respectively. The horizontal axis is the score value (0, 1) and the vertical axis is the sentence number that falls into the corresponding score intervals.

From the score distribution visualisation, it reflects the tuning on pSQM has a larger covered error types while LaBSE is less sensitive to some errors that human experts would spot out. As shown on these charts, pSQM human rating shows much wider "tail" of "low score ratings", while LaBSE rating is much more focused. The reason is that LaBSE similarity model underestimates the severity of errors and error types, while humans analyse the meaning and assign proper error penalties in more diverse setting. As an example, the sentence "The comet did not struck the Earth this time." and "The comet did struck the Earth this time." has very close lexical similarity, but the meaning is very different, in this case “opposite”. LaBSE similarity score would not assign significant penalty to such difference, while human will treat it as a major error. This difference plays a crucial role for reliable translation quality evaluation.

For WMT2021 Metrics Task, we submitted our cushLEPOR system scores for zh=>en and en=>de language pairs, both segment-level and system-level evaluation. The training and development set we used are exact the ones from last section (Section 4). We can not tune our cushLEPOR model parameters on en=>ru language pair from the WMT21 official data, because the human labelled MQM and pSQM scores as validation data that cushLEPOR requires do not exist from last year WMT20 set. Instead, we submitted hLEPOR metric for EN=>RU using the parameter settings in hLEPOR as mentioned in the last section. We carried out evaluation on all four official data-sets: newstest2021 (traditional task), florestest2021 (sentences translated as part of the WMT News translation task), tedtalks (additional sets of sentences translated by WMT21 translation systems in the TED talks domain), and challengeset (synthetic outputs generated specifically to challenge automatic metrics).

In this work, we described cushLEPOR, a customised hLEPOR metric which can be automatically trained and optimised using both human labelled MQM scores, as well as large scale pre-trained language model (LM) LaBSE towards better agreement to human experts level judgements and distilled LM performance respectively, and reducing cost at the meantime, e.g. the manual tuning from hLEPOR and high computational demand from LMs.

We also optimised cushLEPOR towards human translators’ evaluation scores, i.e. pSQM, which showed much improved performance than BLEU and original hLEPOR (with default parameters). Our research is in line with the MT evaluation guideline suggestions from the very recent work Marie et al. (2021) that better evaluation metrics in correlation to human judgement shall be tested and deployed. Or human judgements shall be carried out directly wherever possible.

We have some findings during the experimental investigation: 1) cushLEPOR trained on LaBSE can replace LaBSE to carry out similarity calculation task in MT evaluation, which is much more light weighted and low cost from computational power and complexity point of view. 2) we can choose alternative pre-trained language models (LMs) in the future to boost performance. 3) this cushLEPOR optimisation framework proves to be functional, offering high performance towards pre-trained LMs, much improved agreement of cushLEPOR to LaBSE scores in comparison to hLEPOR (as in Figure 1 and 2). 4) optimised cushLEPOR achieves better agreement towards professional translator’s evaluation (pSQM).

Optuna, the hyper parameter optimisation toolkit we used, can generate different set of cushLEPOR parameter values in different runs, which could be an consistency issue. However, we believe it optimises the performance of cushLEPOR towards the highest agreement to the reference scoring (pre-trained LMs or human evaluations), but not to ensure the same set of parameter values to be generated, so this will not be a problem. We will carry out further analysis on this aspect in the future work.

The hybrid version of hLEPOR Han et al. (2013b) use POS features to function as pseudo synonyms to capture alternative correct translations. However it relays on POS taggers for target language, which does not exist for newly proposed languages, and its tagging accuracy may be low, and it costs extra processing steps. In the future work, we plan to carry out integrated model which combine the POS tagging as a command function in data pre-processing for hybrid cushLEPOR.

Overall, cushLEPOR achieved the first cluster performances in News Domain data on Chinese-English and English-German in WMT2021 Metrics task, while hLEPOR wins TED domain data on English-Russian Freitag et al. (2021). In the future work, we plan to carry out optimisation of cushLEPOR on more language pairs as well as more domains. We will keep our updated parameter set for extended languages and domains available on our cushLEPOR open-sourced platform.

The ADAPT Centre for Digital Content Technology is funded under the SFI Research Centres Programme (Grant 13/RC/2106) and is co-funded under the European Regional Development Fund. We thank the following open-source teams: Google Research for sharing their human evaluation data on MQM and pSQM scores, Optuna open-source hyper-parameter optimisation framework, and LaBSE pre-trained LMs.

The word level hLEPOR default parameters manually tuned for WMT2013 MT evaluation task across language pairs Han et al. (2013a, b) are displayed as below. Both Python (https://pypi.org/project/hLepor/) and Perl (https://github.com/lHan87/aaron-project-hlepor) version codes can be applied to:

On English-to-Czech/Russian (EN=>CS/RU):

• •

  alpha = 9.0,

• •

  beta = 1.0,

• •

  n = 2,

• •

  weight_elp = 2.0,

• •

  weight_pos = 1.0,

• •

  weight_pr = 7.0.

On English-to-German (EN=>DE):

• •

  alpha = 9.0,

• •

  beta = 1.0,

• •

  n = 2,

• •

  weight_elp = 3.0,

• •

  weight_pos = 7.0,

• •

  weight_pr = 1.0.

On Czech / Spanish / Russian to English (CS/ES/RU =>EN):

• •

  alpha = 1.0

• •

  beta = 9.0

• •

  n = 2

• •

  weight_elp = 2.0

• •

  weight_pos = 1.0

• •

  weight_pr = 7.0

On German/French-to-English (DE/FR=>EN) and English-to-Spanish/French (EN=>ES/FR):

• •

  alpha = 9.0

• •

  beta = 1.0

• •

  n = 2

• •

  weight_elp = 2.0

• •

  weight_pos = 1.0

• •

  weight_pr = 3.0