Improving Human-Labeled Data through Dynamic Automatic Conflict Resolution

Annotated natural language data is invaluable to machine learning techniques and algorithms. Two major questions arise as to how to optimize data collection task and how to measure the accuracy and reliability of the resulting data.

Many authors emphasize the importance of careful task design, guideline creation, and pre-testing as seen in [Compton et al., 2012, Compton et al., 2012, Pustejovsky and Stubbs, 2012]. Such tests assist in improving inter-annotator reliability — the extent to which the results of an annotation task represent replicable unbiased judgments. Several reliability estimates correct for this, see especially Krippendorff’s $\alpha$ [Krippendorff, 2004].

Once data is collected, it is standard practice to take Inter-Annotator Agreement (IAA) to provide an estimate of the correctness or validity of human annotations ([Smyth et al., 1995]). IAA is distinct from the reliability metric: the former cannot account for biases that might be shared by a group and hence lead to high agreement across annotators. There is no consensus on how to measure IAA, but one common metric is the Kappa ($\kappa$) statistic (or some variant), which measures pairwise agreement among a set of annotators making category judgments and correcting for expected chance agreement:

$\kappa=\frac{Pr(a)-Pr(e)}{1-Pr(e)}$,

This measure has been shown to be effective, but it also has certain limitations [Eugenio and Glass, 2004, Pustejovsky and Stubbs, 2012, Artstein, 2017]. See also [Craggs and Wood, 2005, Passonneau and Carpenter, 2014, Kara et al., 2018] for some less commonly used measures.

In addition to the ongoing debate about the accuracy and correctness of the $\alpha$ and $\kappa$ statistics as proxies for inter-annotator reliability and agreement respectively, the metrics provide only a partial picture of human-annotated data. The statistic is unable to account for the level of expertise of the annotators, for example. In a meta study, [Bayerl and Paul, 2011] identify several factors that may influence IAA: annotation domain, number of categories in a coding scheme, number of annotators in a project, whether annotators received training, the intensity of annotator training, the annotation purpose, and the method used for calculating IAA. One of their key conclusions is that complex annotation tasks should be simplified to use fewer grading categories. [Bayerl and Paul, 2011] additionally suggest using expert annotators who receive specialized training and collecting a large number of annotations for critical tasks.

The use of expert annotators versus naive ones is a contested issue. While some authors suggest using experts or adjudicators [Kowtko et al., 1993], Snow et al. [Snow et al., 2008] argue that many labeling tasks can be effectively designed and implemented using non-expert annotators recruited via crowd-sourcing tools.

Many solutions exist to resolve the low-quality contributions that come either from deliberate scamming or poor performance [Ipeirotis et al., 2010, Dow et al., 2012, Ipeirotis et al., 2014, Gadiraju et al., 2015]. The most commonly adopted technique for quality assurance relies on a set of questions with known correct answers, often referred to as gold standard test questions, which are randomly injected into the workers’ sessions in order to measure their performance [Checco et al., 2018, Sun and Dance, 2012, El Maarry et al., 2014]. In addition, since the original Dawid-Sekene [Dawid and Skene, 1979] crowdsourcing model based on maximum likelihood estimation, several improvements have been proposed: the Fast Dawid-Skene model for improved speed of convergence [Sinha et al., 2018], the effort to extend to multiclass labeling from binary labeling [Li and Yu, 2014], and the Minmax Entropy method [Zhou et al., 2012] for accuracy improvements. There are also other comparable models [Raykar et al., 2010, Welinder et al., 2010, Karger et al., 2011], but none could outperform Dawid-Skene [Liu et al., 2012].

However, such techniques share several clear limitations:

1. 1.

   Scalability Challenge: the curation of gold standard test questions, or the initialization for ground truth in maximum likelihood estimation methods, often demands significant effort and resources (as many as 30 identical batch of graders per question assignment), and therefore such question sets are usually small in size [Oleson et al., 2011];

2. 2.

   Measurement by Proxy: the gold standard test questions only measure the annotator performances on the given question set, and the results do not necessarily reflect their performances on the actual tasks; further, the question set may differ significantly in terms of style or difficulty from the requests in the actual annotation task, leading to greater deviation from the intended measurement;

3. 3.

   Vulnerability to Exploitation: the inherent limit on the size of the gold standard test question set makes the method prone to a coordinated attack from even a small group of annotation workers, once they can collude and identify the gold questions; Checco et al. [Checco et al., 2018] have proposed an attack mechanism targeting the gold question-based quality assurance method.

This work is in part a continued discussion on inter-annotator (dis)agreement and its implications. As suggested by many [Dumitrache et al., 2018b, Cheatham and Hitzler, 2013, Plank et al., 2014, Hollenstein et al., 2016], the lack of consensus among a group of annotators on a specific task can be attributed to one or more of the following causes: inherent uncertainty in the relevant domain knowledge, true natural language ambiguity, and annotation error. Here we do not attempt to provide a comprehensive interpretation of inter-annotator disagreement. Instead, we propose an annotation strategy that efficiently preserves material inter-annotator disagreement while reducing the impact of annotation errors on consensus extraction. We then share findings from a text classification task which uses this strategy and examine some specific cases of inter-annotator disagreement. We show that the DACR strategy can assist in identifying material disagreements — ones which stem from issues with domain knowledge or the ontology used in the classification task — while reducing disagreements related to simple human error.

The LIEM treats a given annotation process as a black box system with the following assumptions:

• •

  Singular ground truth label: each request has a singular ground truth label;

• •

  Absence of systematic confusion: the annotation process yields incorrect labels from a discrete uniform distribution and doesn’t contain systematic confusions;

• •

  Memoryless behavioral stability: the annotation process is not affected by a priori events, and exhibits stable behavior over time.

Given these assumptions, we may assign labels to a set of requests using the same predetermined annotation process in two duplicate annotation projects, and compute the proportion of requests that received identical labels, $\hat{Y}$. The latent accuracy may then be expressed as:

$\mu_{X}\approx\sqrt{\hat{Y}}$,

In the remainder of this section we explicate this result in detail. To start, consider a project with an unspecified annotation task for a set of requests $R$. Let us assume each request $r\in R$ has a singular ground truth label $g_{r}\in\Omega$, where $\Omega$ is a finite set of all permissible labels in the annotation task.¹¹1Although we acknowledge potential ambiguity and existence of multiple correct labels, this stipulation allows us to study the impact of annotation inaccuracy for the prevalent choice of singular ground truth label in semantic annotation tasks.

Each request $r$ undergoes a predetermined annotation process, which returns a label $l_{r}\in\Omega$. In the crowdsourcing context, the returned label $l_{r}$ is extracted from the answer(s) provided by one or more randomly selected annotation workers. Therefore, if we treat the predetermined annotation process as a black box system and focus on its input-output relationship, the system is expected to behave inconsistently in terms of its performance (that is, its ability to yield the correct answer such that $l_{r}=g_{r}$) due to (1) varying difficulty levels of each request and (2) non-uniformity of performance both amongst annotation workers and for any individual worker over different requests.

Let $X$ be a continuous random variable over $[0,1]$ associated with the performance of the predetermined annotation process. The performance of the system for each request $r$ is defined as the likelihood of yielding the correct label, i.e. $P(l_{r}=g_{r})$. Accordingly, we consider the performance of the system, $p_{r,t}\in[0,1]$, for each request $r$ at time $t$ to be an independent random draw from $X$. We denote the expected performance of the system as $E[X]=\mu_{X}$, the value we intend to estimate.

Next, we briefly discuss two important concepts of probability: the Central Limit Theorem (CLT) and Bhatia-Davis inequality. More complete introductions can be found in [Barron and others, 1986, Bhatia and Davis, 1993]. Without loss of generality, we may treat the performances $\{X_{1},X_{2},...X_{n}\}$ of a black box system over a set of $n$ annotation requests (where $n=|R|$) as a random sample of size $n$ from the given distribution of $X$, with expectation $E[X_{i}]=\mu_{X}$ and variance $Var[X_{i}]=Var[X]=\sigma_{X}^{2}$. Such a set of random variables are independent, identically distributed. Conveniently, then, the sample mean:

$\overline{X}=\frac{1}{n}\sum_{i=1}^{n}X_{i}=\frac{1}{n}\sum_{i=1}^{n}1\cdot X_{i}$,

$\sigma^{2}\leq(M-\mu)(\mu-m)$,

$\sigma_{X}^{2}\leq(1-\mu_{X})\mu_{X}$.

The method to estimate the accuracy of the predetermined annotation process is trivial in the case when $R$ is a gold standard question set so that the proportion of correctly labeled requests is directly observable. Since the premise of this work is to estimate accuracy without relying on gold questions, we continue on to next section.

Turning back to our labeling task, in order to find a reliable proxy variable, we examine the annotation process for a single request $r$ in detail. Given the finite set of all permissible labels in the annotation task, $\Omega$, for request $r$ with ground truth label $g_{r}\in\Omega$, the likelihood of receiving the correct label such that $l_{r}=g_{r}$ is previously defined as the performance of the annotation process on $r$, i.e. $X_{r}\equiv P(l_{r}=g_{r})$.

Accordingly, the probability of not receiving a correct label is $P(l_{r}\in\Omega\setminus\{g_{r}\})=1-P(l_{r}=g_{r})$. Now, let us assume that when the annotation process (treated as a black box system) does not yield the correct label, it simply returns one of the remaining labels in the sample space, i.e. $\Omega\setminus\{g_{r}\}$, with equal probability³³3[Giancola et al., 2018] suggest that mislabeling is not necessarily follow a discrete uniform distribution, and systematic confusions (aptly named ”class permutations”) are observed among the annotation workers on several tasks including text clustering, rare class simulation, and dense image segmentation. However, their findings from the text clustering task also suggest that such systematic confusions are relatively less prevalent in the given task. We assume a discrete uniform distribution as a necessary approximation to avoid intractability..

Consider a request $r$ undergoing the said annotation process twice, with the system’s performances at $X_{r}=p,X_{r}^{\prime}=q$, yielding labels $l_{r},l_{r}^{\prime}$ respectively. With the help of Bayes’ theorem, we may write the probability of receiving two identical labels as:

$P(l_{r}=l_{r}^{\prime})=$
$P(l_{r}=l_{r}^{\prime}|l_{r}=g_{r},l_{r}^{\prime}=g_{r})\cdot P(l_{r}=g_{r})\cdot P(l_{r}^{\prime}=g_{r})+P(l_{r}=l_{r}^{\prime}|l_{r}\neq g_{r},l_{r}^{\prime}\neq g_{r})\cdot P(l_{r}\neq g_{r})\cdot P(l_{r}^{\prime}\neq g_{r})$,

from which we calculate:

$P(l_{r}=l_{r}^{\prime})=p\cdot q+\frac{1}{m-1}\cdot(1-p)(1-q)$,

Further, with reasonably good faith in the system performances such that $pq\gg(1-p)(1-q)$ and a sufficiently large sample space (e.g. $m>10$), we may allow for further approximation given a significantly smaller second term⁴⁴4Essentially, the chances of receiving identical labels from two independent annotation sessions on the same request are negligible when both labels are incorrect. as $\frac{1}{m-1}\cdot(1-p)(1-q)\rightarrow 0$ and $p\cdot q\gg\frac{1}{m-1}\cdot(1-p)(1-q)$. This approximation enables us to simplify the expression for $P(l_{r}=l_{r}^{\prime})$ to:

$P(l_{r}=l_{r}^{\prime})=p\cdot q$,

$Y_{R}=X_{R}X_{R}^{\prime}$,

To measure consistency at scale, we set up two annotation projects with an identical set of requests $R$. This yields two sets of labels, $L=\{l_{1},l_{2},...,l_{n}\},L^{\prime}=\{l_{1}^{\prime},l_{2}^{\prime},...,l_{n}^{\prime}\}$. We may then treat the consistency observations $\{Y_{1},Y_{2},...,Y_{n}\}$ as a random sample of size $n$ from a distribution $Y$, with expectation $E[Y]=\mu_{X}^{2}$ and variance $Var[Y]=(2\mu_{X}^{2}+\sigma_{X}^{2})\sigma_{X}^{2}$. Assuming that they are independent, identically distributed, according to Central Limit Theorem the distribution of sample mean converges to a normal distribution when the sample size is large enough:

$\overline{Y}\overset{\mathrel{D}}{\longrightarrow}\mathcal{N}(\mu_{X}^{2},(2\mu_{X}^{2}+\sigma_{X}^{2})\sigma_{X}^{2}/n)$,

$Var[\overline{Y}]=(2\mu_{X}^{2}+\sigma_{X}^{2})\sigma_{X}^{2}/n\leq\frac{33}{64n}$,

Meanwhile, the established upper bound helps us to properly capture the margin of error in estimating $E[Y]$ for a not-so-large size of $n$. For example, with $n=100$ we have $Var[\overline{Y}]\leq 5.16\cdot 10^{-3}$; to maintain a confidence level of a three-sigma effect (99.73% confidence) we would estimate $\mu_{X}^{2}$ to be $\overline{Y}\pm 0.015$.

In sum, the Latent Inaccuracy Estimation Method treats a given annotation process as a black box system and assumes that (a) each request has a singular ground truth label; (b) the annotation process yields incorrect labels from a discrete uniform distribution and doesn’t contain systematic confusions; and (c) the annotation process is not affected by a priori events, and exhibits stable behavior over time.

Therefore, if we assign labels to a set of requests using the same annotation process in two duplicate projects, we may compute the proportion of requests that receive identical labels, $\hat{Y}$. The latent accuracy of the annotation process may then be expressed as: $\mu_{X}\approx\sqrt{\hat{Y}}$, with decreasing margin of error for larger values of $n$.

We compare the following three commonly used annotation strategies:

1. 1.

   One-grader: each request is assigned to a randomly selected grader and the answer label is recorded;

2. 2.

   Double Graded, Conflict Resolved (DG, CR): each request is first assigned to two randomly selected graders; if the two agree, the consensus answer label is recorded; otherwise (in the case of a conflict), the request is escalated to an expert annotator with higher expected performance, and the expert’s answer label is recorded as the final answer;

3. 3.

   N-Graded: each request is assigned to N randomly selected graders; the answer label with a simple majority (greater than $\frac{1}{2}$) is recorded as the final answer. If no answer label with a simple majority exists, the request is assigned an in-conflict label.

Against these approaches, we propose a novel approach: Dynamic Automatic Conflict Resolution (DACR) with maximum $N$ grades: each request is first assigned to two randomly selected annotation workers; if the two agree, the consensus answer label is recorded; otherwise (in the case of a conflict), the request is iteratively assigned to one additional worker until (a) a simple majority answer label (from all historical answers) is produced, or (b) the maximum $N$ grades is reached and the annotation process ends (regardless of the presence of a majority label). A separate process can then apply to determine how to resolve in-conflict requests.

Our simulation experiments vary the following parameters of the crowdsourcing annotation task: (1) the quality of the annotation workers, (2) the quality of expert annotators (if any), (3) the difficulty levels of the requests in the annotation task, and (4) the annotation strategy in use.

As detailed in Table 1, the parameters are initialized⁶⁶6We also simulated with varying parameter values and the subsequent findings are established consistently. with reasonable estimates from empirical observations. Accordingly, the annotation act is abstracted as follows: when an annotator with capability $c$ annotates a request with difficulty level $d$, the correctness of the answer label is generated as the outcome of a single Bernoulli trial, with $P(correct)=d\cdot c$, and $P(incorrect)=1-(d\cdot c)$.

Our simulation experiments investigate (1) the performance and efficiency of the competing annotation strategies outlined above, and (2) the validity of the proposed LIEM approach in estimating the expected performance of a given annotation strategy.

We ran 100 simulated annotation projects for each annotation strategy; each project consisted of the same batch of randomly initialized 10,000 requests. We plot the distribution of the annotation accuracy over the projects for each strategy in Figure 1.

We conclude in Table 2 that the DACR strategy is consistently ranked top 2 compared to its competitors along all three metrics listed. We furthermore find that the DACR strategy requires a significantly lower number of grades to achieve comparable performance to that produced by a 7-graded strategy (63.6% lower than this top competitor), and achieves improved performance with a small incremental number of grades compared to the $DG,CR$ strategy (9.0% higher accuracy compared to this top competitor).

To validate the LIEM approach, we exhaustively compute the inter-project consistency rates and the corresponding pair’s mean accuracy, for each annotation strategy. The data points are plotted in Figure 2, where we conclude with simple curve fitting that LIEM’s proposed estimation method, $\mu_{X}\approx\sqrt{\hat{Y}}$, is a reasonably accurate approach to find the latent accuracy for a variety of annotation strategies.

Having examined DACR’s (superior) performance via simulation experiments, we deployed it to our internal crowdsourcing platform for annotation on a text classification task. The project consisted of 60,000 randomly sampled utterances and used the DACR annotation strategy, with the minimum grade count set to 3 and the maximum set to 9. At the completion of the project, we recorded an average of 3.74 grades per utterance. Additionally, 5.00% of the utterances did not receive a consensus label even when the maximum grade count of 9 had been reached (we will refer to such cases as “in-conflict”).⁷⁷7In simulation, we record 2.55 average grades per utterance with $\mathbf{Min=2}$ and $\mathbf{Max=5}$ , and an in-conflict rate of 5.1%.

To determine whether such in-conflict requests are a result of inherent ambiguity or poor performance of annotation workers, we construct a confusion matrix based on the inter-annotator disagreements over the set of requests with no consensus label. As we observe from Figure 3, although we omit the label names to preserve privacy — there is a clear pattern where a small number of specific intent labels are significantly more likely to be confused with a small set of select other labels.⁸⁸8Notice the existence of highlighted blocks in Figure 3. Such highlighted blocks correspond to unusually high levels of inter-category confusion, and are to be rarely observed in the confusion matrix if no systematic bias exists. The confusion matrix is normalized by adjusting for background frequency of each category.

Accordingly, the observed systematic bias indicates the presence of one or several of following factors: (1) ambiguity or vagueness in the annotation guidelines, (2) inherent ambiguity in the ontology design, or (3) inherent ambiguity in a few select types of utterances.

An examination of the in-conflict cases has led us to conclude that in this study, (2) and (3) were sizable contributing factors. As a result, we have been able to undertake a series of directed efforts aimed at educating the annotators on the one hand, and better-defining the ontology on the other, to reduce conflicts caused by reason (2). On the other hand, we do not expect that any effort on our part can resolve conflicts casued by reason (3). Instead, the prevalence of truly ambiguous utterances in our study provides yet more evidence that the common insistence on a single ground truth label for each request may simply be misguided. In other words, such an assumption makes it inherently impossible to correctly represent 100% of natural language utterances.

Acknowledging the existence of ambiguity has two profound implications for nearly all annotation tasks that conforms to the premises of this study:

• •

  Information loss: as [Dumitrache et al., 2018a] have previously suggested, the rejection of ambiguity results in a loss of information that would impact a wide range of supervised learning applications and prevents the development of appropriate model architectures;

• •

  Measurement error: we find it equally concerning that if we are to generate a test set with annotation strategies that ignores ambiguity (such as one-grader, or DG,CR), we would have deliberately introduced noise into the test set by introducing ambiguous examples with an uncertain partial label. This omission would lead to increased measurement error on model performance.⁹⁹9Consider a test set with 5% ambiguous examples for which no (single) ground truth exists. Then even the oracle would be measured to have a 5% error rate on this test set.