Adaptive Crowdsourcing Via Self-Supervised Learning

Aggregating opinions from a diverse group often yields more accurate estimates than relying on a single individual. Applications are wide-ranging. Intelligence agencies query crowds to help predict uncertain and significant events, showing that a collective effort can give better results (Tetlock and Gardner, 2016). Aggregating inputs has also been shown to improve answers to questions in the single-question crowd wisdom problem (Prelec et al., 2017) and in financial forecasting (Da and Huang, 2020). Modern artificial intelligence (AI) also relies on aggregation of input from human annotators for the development of image classifiers (Vaughan, 2017) and chatbots (Ouyang et al., 2022; OpenAI, 2023; Touvron et al., 2023; Google, 2023; Anthropic, 2023).

The recent explosion of AI has led to considerable discussion of situations where computers could partly or wholly replace humans for crowdsourcing tasks (Zhu et al., 2023; Törnberg, 2023; Ollion et al., 2023; Boussioux et al., 2023). The algorithms we discuss in this paper could apply to such settings but were conceived to apply to groups of people in the spirit of initial work on crowdsourcing. The mid-nineteenth-century Smithsonian Institution Meteorological Project processed localized data into national weather maps. One hundred fifty volunteer respondents provided information by telegraph (Smithsonian Institution Archives, 2024), which was processed by a dozen-plus humans to create a national data map. By 1860, five hundred local weather stations were reporting (National Weather Service, 2024).

Many traditional crowdsourcing examples involve large numbers of participants. However, the underlying principles apply even with quite small groups, as would be the norm in many contexts, such as with boards of directors or R&D research teams.

A conventional approach to aggregation assigns equal importance to each crowdworker’s input. This approach, which we refer to as averaging, is simple and enjoys wide use. With a sufficiently large number of crowdworkers, averaging performs well. However, engaging so many crowdworkers can be costly or infeasible. Because of this, practical crowdsourcing systems work with a limited number of crowdworkers, and averaging leaves substantial room for improvement. We discuss two simple cases where averaging can require too many crowdworkers in order to produce accurate estimates. One arises when skills vary across crowdworkers; those who are more skilled ought to be assigned greater weight. Another example arises when crowdworkers share sources of information, perhaps because they watch the same news channels or follow similar accounts on social media, or utilize the same data sets. In this case, each crowdworker’s estimate does not provide independent information, and the crowdworker’s weight should decrease with the degree of dependence.

In principle, if each crowdworker’s level of skill and independence are known, it should be possible to derive weights that improve group estimates relative to averaging. Our work introduces an approach that approximates these weights given estimates of past outcomes provided by the same crowdworkers. The weights reflect what is learned about skills and independence.

If patterns among crowdworker estimates are expressed by simple – for example, linear – models, existing algorithms offer effective means for fitting to past observations. Expectation maximization (EM), in particular, offers a popular option used for this purpose (Zhang et al., 2016a; Zheng et al., 2017). However, in many contexts, the relationships are complex and call for flexible machine learning models such as neural networks. Computational requirements of existing algorithms such as EM become onerous.

To address the limitations of existing methods, we introduce a new approach called predict-each-worker, which leverages self-supervised learning and a novel aggregation scheme. This approach uses self-supervised learning (SSL) (Liu et al., 2021) to infer patterns among crowdworker estimates. In particular, for each crowdworker, predict-each-worker produces an SSL model that predicts the crowdworker’s estimate based on past observations and estimates of other crowdworkers. This enables modeling of complex patterns among crowdworker estimates, for example, by using neural network SSL models. The aggregation scheme leverages these SSL models to weight crowdworkers to a degree that increases with the crowdworker’s skill and independence.

In short, the predict-each-worker approach employs each individual crowdworker’s estimates, after processing by the center, as the basis for producing a group estimate. These individual estimates represent an atomic approach for building a group estimate. This approach contrasts sharply with other methods that instead proceed from a series of group estimates. For example, the famed Delphi method (Dalkey and Helmer, 1963) involves a panel of experts or crowdworkers who, through several rounds of interactions, refine their estimates. After each round, a facilitator provides an anonymous summary of the crowdworkers’ estimates. The crowdworkers are then encouraged to revise their earlier estimates in light of the replies of other members of their panel. This process is repeated until the group converges on a consensus. This method has been applied, for example, to military forecasting (Dalkey and Helmer, 1963), health research (de Meyrick, 2003), and as a method to do graduate research (Skulmoski et al., 2007). While problem settings where crowdworkers could update their estimates are important and interesting, our approach, predict-each-worker, is not designed for such settings, and we leave it for future work to develop an extension.

To investigate the performance of our approach, we perform both computational and theoretical studies. For a Gaussian data-generating process, we prove that our algorithm is asymptotically optimal, and through simulations, we find that this method outperforms averaging crowdworker estimates. Moreover, our empirical results indicate that when the dataset is large, predict-each-worker performs as well as an EM-based algorithm. These studies offer a ‘sanity check’ for our approach and motivate future work for more complex data-generating processes as well as real-world problem settings.

We consider a crowdsourcing system as illustrated in Figure 1. In each round, indexed by positive integers $t$, $K$ distinct crowdworkers provide estimates $Y_{t}\in\Re^{K}$ for an independent outcome $Z_{t}\in\Re$. A center observes estimates $Y_{t}$ to produce a group estimate $\hat{Z}_{t}\in\Re$ for the outcome $Z_{t}$. We assume that the center does not observe the outcomes.

To model uncertainty from the center’s perspective, we take $Z_{t}$ and $Y_{t}$ to be random variables. These and all random variables we consider are defined with respect to a probability space $(\Omega,\mathbb{F},\mathbb{P})$. We assume that $(Y_{t}:t\in\mathbb{Z}_{++})$ is exchangeable, and for simplicity, we assume that $\mathbb{E}[Y_{t}]=0$.

While we could consider such a process for any sequence of outcomes $Z_{t}$, we will restrict attention to the case where $Z_{t}$ is the consensus of an infinite population of crowdworkers, in a sense we now define. For each $t$, we model crowdworker estimates $(\tilde{Y}_{t,k}:k\in\mathbb{Z}_{++})$ as an exchangeable stochastic process. We assume that the limit $\lim_{K\to\infty}\sum_{k=1}^{K}\nicefrac{{\tilde{Y}_{t,k}}}{{K}}$ exists almost surely and the consensus is given by

We define $Z_{t}$ this way because we believe that such a consensus estimate is often useful – this is motivated by work on the wisdom of crowds (Larrick and Soll, 2006). Another benefit of such a definition is that one can assess performance using observable estimates from out-of-sample crowdworkers without relying on subjective beliefs about the relationship between estimates and outcomes. The notion of assessment based on out-of-sample crowdworker estimates can be used for hyperparameter tuning, as explained in Appendix D.5.

We take $Y_{t}$ to be the first $K$ components $\tilde{Y}_{t,1:K}$. Hence, each of the $K$ crowdworkers can be thought of as sampled uniformly and independently from the infinite population. The center’s objective is to produce an accurate estimate of the consensus $Z_{t}$ based on estimates supplied by the finite sample of $K$ crowdworkers.

Estimates $Y_{t}$ depend on unknown characteristics of the crowdworkers, such as their skills and relationships. By de Finetti’s Theorem, exchangeability across $t$ implies existence of a random variable $\theta$ such that, conditioned on $\theta$, $(Y_{t}:t\in\mathbb{Z}_{++})$ is iid. Intuitively, this latent random variable $\theta$ encodes all relevant information about the crowdworkers and their relative behavior. While multiple latent variables can serve this purpose, we take $\theta$ to be one that is minimal. By this we mean that $\theta$ determines the distribution $\mathbb{P}(Y_{t}\in\cdot|\theta)$ and vice versa.

Before producing a group estimate $\hat{Z}_{t}$, the center observes past crowdworker estimates $Y_{1:t-1}$ as well as current estimates $Y_{t}$. The manner in which the center produces its estimate can be expressed in terms of a policy, which is a function $\pi$ that identifies a group estimate $\pi(Y_{1:t})$ based on the observed history $Y_{1:t}$ of crowdworker estimates. When particulars about the policy under discussion matter or when we consider alternatives, we express the dependence of group estimates on the policy using superscripts. In particular, $\hat{Z}_{t}^{\pi}=\pi(Y_{1:t})$. For each $t$, our objective is to minimize the expected loss $\mathbb{E}[\mathcal{L}(Z_{t},\hat{Z}^{\pi}_{t})]$ for some loss function $\mathcal{L}$. With real-valued group estimates, the squared error $\mathcal{L}(Z_{t},\hat{Z}_{t}^{\pi})=(Z_{t}-\hat{Z}^{\pi}_{t})^{2}$ offers a natural loss function.

This section formally introduces our approach, predict-each-worker. We will do so in three steps. First, we present the algorithm in an abstract form, which is not efficiently implementable but applies across real-valued estimate distributions. Then, we describe an implementable instance specialized to a Gaussian data-generating process. Finally, we explain how our approach extends and scales to complex settings.

To understand how our algorithm compares to benchmarks, we perform a simulation study. We use a Gaussian data-generating process and three benchmarks: averaging, an EM-based policy, and the clairvoyant policy. The section first introduces the data-generating process, followed by the three policies, and finally presents the simulation results.

Our literature review is structured around two main categories of prior work. Initially, we examine research directly related to our problem setting, where true outcomes are not observed, and a center interacts with the same crowdworkers multiple times. We aim to succinctly outline the range of algorithms developed for this setting and delineate how these differ from our proposed method, predict-each-worker. For a more comprehensive treatment of this problem setting, see the survey papers by Zhang et al. (2016a); Sheng and Zhang (2019); Zhang et al. (2022). Following this, we address research pertaining to different but related settings, such as those where true outcomes could be observed or interaction with crowdworkers is limited to a single round. This part of the review highlights that our assumptions do not hold for all problem settings of interest and recognizes methods more suited to these alternative settings. Such a comparison is useful, as it informs the reader about the boundaries of applicability of our method and aids in assessing its suitability for different application contexts.

Our problem setting. One of the most popular approaches for this problem setting is to posit a probabilistic model that captures crowdworker patterns and then approximate the maximum a posteriori (MAP) estimates of the parameters that characterize this probabilistic model. The EM algorithm is one way to approximate these MAP estimates and has been applied to a variety of problems. Li et al. (2014), Kara et al. (2015), and Cai et al. (2020) use EM to learn a noise covariance matrix for the case where crowdworker estimates are continuous whereas Dawid and Skene (1979) and Zhang et al. (2016b) use EM to learn a so-called confusion matrix for the case where crowdworker estimates are categorical. Other algorithms to approximate the MAP estimates have been considered in (Moreno et al., 2015; Bach et al., 2017; Li et al., 2019). These algorithms estimate correlations when crowdworker estimates are categorical. Some other works assume that linear models sufficiently capture patterns between crowdworkers conditioned on relevant contexts like images or text (Raykar et al., 2010; Rodrigues and Pereira, 2018; Li et al., 2021; Chen et al., 2021). The algorithms proposed in these works can be interpreted as estimating the MAP estimates of the parameters of crowdworker models conditioned on the context. However, unlike our approach, all the algorithms discussed here are either difficult to extend or become computationally onerous if representing complex patterns requires flexible machine learning models like neural networks.

Cheng et al. (2022) consider fitting parameters of a stylized model that characterizes skills and dependencies among crowdworkers and then using the model to improve group estimates. They demonstrate that resulting group estimates can turn out to be worse than averaging if the model is misspecified. This casts doubt on the practicality of approaches that aim to improve on averaging. Predict-each-worker overcomes this limitation by enabling the use of neural networks, which behave similarly to nonparametric models and thus do not suffer from misspecification to the extent that stylized models do.

Other problem settings. Some prior works consider a problem setting where some or all the outcomes could be observed. Tang and Lease (2011), Atarashi et al. (2018), and Gaunt et al. (2016) develop so-called semi-supervised techniques to improve aggregation for machine learning problems. Another line of work (Budescu and Chen, 2015; Merkle et al., 2017; Satopää et al., 2021) focuses on generating improved forecasts of consequential future events. By having the ability to observe some or all true outcomes, these works propose algorithms that make fewer restrictive assumptions about how crowdworkers relate. Hence, for settings where ground truth information is available for a large number of crowdsourcing tasks, methods in these prior works could be more appropriate, but if ground truth information is sparse, then our approach predict-each-worker could be more helpful.

Now, we discuss a setting where the center only gets to interact with crowdworkers for one round. To gauge the skill level in this setting, prior works (Prelec et al., 2017; Palley and Soll, 2019; Palley and Satopää, 2023) let each crowdworker not only guess the true quantity but also guess what the mean of the other crowdworkers’ estimates will be. The intuition is that crowdworkers who can accurately predict how others will respond will tend to be more skilled. In problem settings where each crowdsourcing task is very different from the others, using methods in the works above might be more beneficial, but if crowdsourcing tasks are similar and allow generalization, then our method could perform better.

While we treat the mechanism for seeking crowdworker estimates as given, there are several works that try to improve the method for collecting crowdworker estimates. We believe that the suggestions given in these works should be used when applying our method. There are three bodies of work that we discuss. First, works by Da and Huang (2020) and Frey and Van de Rijt (2021) suggest that crowdworkers should not get access to estimates of others before they generate their own estimates. These works, using theoretical models and experiments, show that if this requirement is not met, errors made by crowdworkers can covary a lot and may not average out to zero, even with many crowdworkers. As our work relies on the usefulness of averaging, it may be necessary to blind crowdworkers to each other’s estimates.

Second, most crowdworkers get paid. Accordingly, some studies examine how monetary incentives can raise the quality of crowdworkers’ estimates. The work by Mason and Watts (2009) observed that increasing the pay did not increase the amount of quality but only increased the number of tasks that were done. Later, the work by Ho et al. (2015) pointed out that the observation made by Mason and Watts (2009) is usually true when the tasks are easy but would not hold if tasks are “effort responsive.” Their experimental study verified this claim.

Third, works (Faltings et al., 2014; Eickhoff, 2018; Geiger et al., 2020) discuss how cognitive biases can negatively affect the quality of crowdsourced data. To detect and mitigate such biases, the work by Draws et al. (2021) proposes a checklist that designers of crowdsourcing tasks should use.

In this work, we proposed a novel approach to crowdsourcing that we call predict-each-worker. We showed that it performs better than sample averaging, matches the performance of EM with a large dataset, and is asymptotically optimal for a Gaussian data-generating process. Additionally, we discussed how our algorithm can accommodate complex patterns between crowdworkers using neural networks. Taken together, our algorithm’s performance and flexibility make it an attractive alternative to EM and other prior work.

Our work motivates many new directions for future work. First, our aggregation rule is limited to cases where the estimates take continuous values. We leave it for future work to develop simple aggregation rules that can be applied when the estimates are categorical, thereby making our approach directly applicable to many machine-learning problems like supervised learning (Zhang et al., 2016a) and fine-tuning language models (Ouyang et al., 2022). It would also be interesting to extend our approach to handle cases where estimates take complex forms, like in the single-question crowdsourcing problem (Prelec et al., 2017) or when they take forms like text or bounding boxes (Braylan and Lease, 2020). Second, for simplicity and clarity, we restricted our focus to simulated data. We leave it for future work to conduct a more thorough experimental evaluation using real-world datasets.

This work was partly inspired by conversations with Morteza Ibrahimi and John Maggs. The paper also benefited from discussions with Saurabh Kumar and Wanqiao Xu. We gratefully acknowledge the Robert Bosch Stanford Graduate Fellowship, the NSF GRFP Fellowship, and financial support from the Army Research Office through grant W911NF2010055 and SCBX through the Stanford Institute for Human-Centered Artificial Intelligence.