Improving Statistical Significance in Human Evaluation
of Automatic Metrics via Soft Pairwise Accuracy

Automatic metrics are crucial because researchers and practitioners in NLP typically can’t afford the high cost and latency of high-quality human evaluations. Despite their shortcomings, metrics like word error rate and BLEU Papineni et al. (2002)—in conjunction with carefully curated test sets—have been crucial for the field of NLP, as they have provided a yardstick to make continual progress over many decades in automatic speech recognition and machine translation (MT), respectively.

Reliance on automatic metrics makes selecting a good automatic metric of paramount importance. Conceptually, an automatic metric should emulate human judgments. Selecting an automatic metric typically entails generating a set of human judgments for a wide variety of outputs from a large number of different systems, and selecting the automatic metric that produces scores most similar to the human judgments. But how do we quantify similarity? To select the metric which produces output most similar to human judgements, we need a meta-metric to compare metric scores and human judgments. Despite nearly two decades of research on MT meta-evaluation, the community has not reached a consensus on the choice of a meta-metric. Various meta-metrics have been introduced over the years to address problems with prior meta-metrics, while sometimes creating new problems or re-introducing old ones (see § 5).

Recent works Mathur et al. (2020b); Kocmi et al. (2021) have argued that the primary application of a metric is to choose between two competing systems, therefore the best metric is the one which produces pairwise system rankings most similar to the pairwise system rankings produced by human judgements. This led to Pairwise Accuracy (PA) being adopted by the WMT Metrics shared task for the past several years (Freitag et al., 2021, 2022, 2023). However, this argument omits a key detail: standard best practice when comparing two systems with an automatic metric is to consider not only which system the metric prefers, but also whether or not that preference is statistically significant Koehn (2004). Thus we argue that metrics should emulate not only the accuracy of human pairwise ranking, but also the confidence or statistical significance of the human pairwise ranking.

To this end, we propose Soft Pairwise Accuracy (SPA), a new meta-metric which takes into account statistical significance of both the metric scores and the human judgments when evaluating the extent to which the metric in question agrees with the human judgments. We show that soft pairwise accuracy, as its name implies, can be viewed as a soft (i.e. non-binarized) version of PA, and present analysis that demonstrates SPA has several distinct advantages over PA. First, we find SPA is more stable with respect to the exact choice of MT systems and segments used. Second, we show that due to the binarization in its formulation, PA can only assign a small set of distinct output values to metrics, and in practice this results in many metrics being artificially assigned the exact same PA score. We demonstrate that SPA fixes this issue. Finally, we argue that PA is effectively equivalent to SPA with added noise due to binarization. We show that removing this noise (i.e. switching to SPA) results in substantially more statistically significant comparisons between metrics, making SPA a more discriminative and therefore more useful meta-metric. Our findings resulted in SPA being selected as the official system-level meta-metric for the 2024 WMT Metrics Shared Task Freitag et al. (2024).

We propose a simple meta-metric for evaluating automatic metrics given human judgments, which we denote Soft Pairwise Accuracy:

where $N$ is the number of systems for which we have human judgements and metric scores, $p_{ij}^{h}$ is the $p$-value for hypothesis that system $i$ is better than system $j$ given the human judgments, and $p_{ij}^{m}$ is the $p$-value for hypothesis that system $i$ is better than system $j$ given the metric scores. The term ${{N\choose 2}}^{-1}=\frac{2}{N(N{-}1)}$ normalizes the summation by the total number of pairs of systems being compared.

For each pairwise system comparison, we use a permutation test Fisher (1935) to estimate statistical significance of the difference in the means of the segment-level scores from a particular metric (or the human judgements) for the two systems. We first randomly split the segment-level scores (ignoring the labels, i.e. which MT system produced each segment) into two parts and compute the difference in metric score mean. Repeating this process many times provides a set of mean differences we can reasonably expect under the null hypothesis that the two systems are of the same quality. We compute a one-tailed $p$-value by calculating the fraction of the time that the random splits produce differences greater than or equal to the mean difference we observe for the two systems.

Permutation tests are appealing because they don’t require any assumptions about the underlying distribution of the data. This fits our use case well because we cannot assume anything about the distribution of segment-level scores of a metric.¹¹1Metric and human annotation distributions are both highly variable Lo et al. (2023b); Knowles and Lo (2024). Permutation tests instead have the assumption of exchangeability Pitman (1937); Draper et al. (1993); Good (2002)—that is, under the null hypothesis (in our case, that the two MT systems are of equal quality) the joint distribution of the observations is invariant under permutations of the data labels. To help ensure exchangeability, we perform permutations such that each split has exactly one translation of each test set sentence, commonly referred to as a paired permutation test Good (2013).

Here we present some concrete examples for the sake of intuition. Suppose a metric reports a $+10$ point difference between system $i$ and system $j$, and that the random permutations only produce a metric difference $\geq 10$ points $1\%$ of the time. Thus $p_{ij}=0.01$ and we conclude that the metric has high confidence that system $i$ is better than system $j$. Likewise, if the metric reports the systems have a $-10$ point difference, we might find that the random permutations produce a metric difference $\geq-10$ points $99\%$ of the time. Thus $p_{ij}=0.99$ and we conclude the metric has high confidence that system $i$ is worse than system $j$. If the systems have the same metric score, we would expect about half of the random permutations to produce a metric difference $\geq 0$ and thus $p_{ij}=0.5$, indicating the metric finds the two systems indistinguishable from each other.

Meta-metric evaluation is challenging because there is no ground truth (i.e., we don’t know the true ranking of the metrics). Instead, we conduct analysis to compare SPA and PA. First, we study how sensitive the meta-metric results are when ablating the number of MT systems and number of segments per MT system, with the assumption that lower sensitivity to the exact systems/segments used indicates a better meta-metric. Second, we examine whether PA indeed has the problem of ties that we hypothesized in § 2.2, and whether SPA fixes this issue. Finally, we test our hypothesis that the binarization in PA is effectively acting as additive random noise, and that SPA is effectively the same underlying meta-metric with the noise term removed.

WMT has run a machine translation evaluation since 2006 Koehn and Monz (2006). Since 2007 Callison-Burch et al. (2007), there has also been meta-evaluation of automatic metrics on the submitted translation systems. Here we summarize the rich 17 year history of system-level meta-evaluation at the WMT Metrics Shared Tasks⁶⁶6The WMT Shared Tasks have typically evaluated at both the system- and segment-level, but we focus on system-level meta-evaluation as it is most relevant to our work. and work related to and directly impacting the shared tasks, in order to demonstrate how our work fits into the historical context.

In the WMT 2007-2013 metrics evaluations Callison-Burch et al. (2007, 2008, 2009, 2010, 2011, 2012); Macháček and Bojar (2013) Spearman’s rank correlation coefficient $\rho$ was used for meta-evaluation of metrics. This was motivated by the fact that Spearman’s makes fewer assumptions about the data than the Pearson correlation coefficient.

The WMT 2013 Translation Shared Task Bojar et al. (2013) introduced system clusters (groups of systems that cannot be distinguished given the human judgments), and the 2013 metrics task Macháček and Bojar (2013) introduced empirical confidence of Spearman’s $\rho$ using bootstrap resampling. Since they were not able to resample on the submitted metrics, they only re-sampled human judgments. This iteration also discussed the fact that Spearman’s $\rho$ does not give partial credit. The penalty is equal for all wrong judgments, regardless of if the systems are close or far in quality. To compensate they present additional methods of analysis: Pearson’s, and correlation with systems’ clusters from the translation task Bojar et al. (2013). Those clusters were treated as ‘ranks with ties,’ and then correlation computed against Pearson’s and Pearson’s correlation against ‘fuzzy ranks’ (the average over ranks of all systems that are not significantly different in human quality).

In 2014, the metrics task Macháček and Bojar (2014) fully switched to Pearson’s $r$ from Spearman’s $\rho$. They also did bootstrap resampling to get empirical confidence intervals of system level correlations. This change to Pearson’s was due to the concerns pointed out in the previous year’s shared task, which had explored other meta-metrics.

The 2015 metrics task Stanojević et al. (2015) continued with Pearson’s $r$, and also presented analysis of Pearson’s $r$ vs Spearman’s $\rho$, and highlighted the instability of Spearman’s $\rho$ when MT systems are similar.

The 2016 metrics task Bojar et al. (2016) stuck with Pearson’s $r$, but changed the confidence to be the Williams test Williams (1959), as Graham and Baldwin (2014) had noted that this test is appropriate for dependent correlations.

The 2017 metrics task Bojar et al. (2017) kept Pearson’s $r$, and Williams test. They also added a pairwise significance test using Williams test. This continued in 2018 and 2019 Ma et al. (2018, 2019)

The 2020 metrics task Mathur et al. (2020b) continued to use Pearson’s, but also includes Kendall’s Tau for analysis. Kendall’s Tau is a closer match for the system ranking use case, since it is evaluating whether the ordering of a pair of systems is the same as the human ordering. However, it does not take into account the magnitude difference.

In 2021, the metrics task Freitag et al. (2021) adopted pairwise accuracy Kocmi et al. (2021), motivated in part by the fact that MT system outliers had an outsized impact on Pearson correlation when it is used to rank MT Metrics Mathur et al. (2020a). Pairwise accuracy produces the same system-level ranking as Kendall’s Tau, as they are equivalent modulo a linear scaling and shifting (see § 5.1). The PERM-BOTH hypothesis test of Deutsch et al. (2021) was used to determine significance. 2021 and 2022 Freitag et al. (2022, 2023) follow.

In summary, the historical context of the WMT metric evaluations demonstrates that meta-evaluation is very challenging due to the numerous issues that must be simultaneously addressed, and underscores the pitfalls of making changes to meta-evaluation without considering the full set of ramifications. Most relevant to our work, it appears that the switch to pairwise accuracy in 2021 reduced the influence of outliers Mathur et al. (2020a) and (somewhat) aligned meta-evaluation with the standard use of comparing two systems with a metric, but it also reintroduced a problem that was first pointed out by Macháček and Bojar (2013) and more fully addressed by the change to Pearson’s $r$ from Spearman $\rho$ by Macháček and Bojar (2014): a disregard for the magnitude of differences. We address this issue by considering empirical confidence, which was first added by Macháček and Bojar (2013), and in the process we also better align meta-evaluation to the (more correct) usage of comparing two systems with a metric while also considering the statistical significance of the results.

We introduce a new meta-metric which we denote soft pairwise accuracy, and show that it improves on pairwise accuracy in a number of ways, most notably that it is more stable than pairwise accuracy when ablating the number of systems and annotations per system, it fixes an issue of metric ties observed in pairwise accuracy, and it produces more statistically significant comparisons between metrics than pairwise accuracy. We also discuss how soft pairwise accuracy fits into and builds upon the nearly two decade history of meta-evaluation at the WMT Metric Shared Tasks.

This work would not have been possible without the WMT shared tasks and metrics tasks releasing their data. We are appreciative of helpful discussions with the WMT metrics task organizers as well as Sweta Agrawal and Rebecca Knowles.

When computing $p$-values, we assume that system-level metric scores are the average of segment-level metrics scores. There is a line of recent work that seeks to incorporate contextual information into automatic metrics. Many such works still produce scores at the segment level (e.g. Vernikos et al., 2022; Hu et al., 2023; Agrawal et al., 2024) but others produce one score per window of a few sentences Raunak et al. (2024) or one score per paragraph Deutsch et al. (2023b). Our method should still be applicable in such cases, but would require permuting windows or paragraphs instead of segments. Additionally, as previously noted, some metrics—notably BLEU Papineni et al. (2002) and chrF Popović (2015)—compute statistics at the segment level and combine them to create document-level scores. Again, permutations would still work but would require some modification. To the best of our knowledge, this issue is not limited to our work—the same assumption is made in prior work computing statistical significance of metrics, including the WMT shared tasks Freitag et al. (2021, 2022, 2023) and Deutsch et al. (2021).

It is worth noting that the permutations in this work (as in prior works) are done on a single test set, and do not necessarily reflect variations in performance that could result from using the metrics in another domain. Prior work has shown that trained metrics are sensitive to a shift in domain relative to the data domain they were trained on Zouhar et al. (2024).