Collective inference of the truth of propositions
from crowd probability judgments

We first consider the accuracy achieved by simple methods for aggregating the judgments. The accuracy of an individual person or collective inference algorithm is defined as the rate of correct answers. An estimated probability is counted as correct if it falls on the right side of 0.5. If we chose a single peer’s probability rating at random, the accuracy of our collective inferences about the claims would match the average accuracy of the peers: about 62% for the set of 1,200 general-knowledge claims in our online experiment (Fig. 2). Trusting a random peer does not benefit from the wisdom of the crowd.

A simple method to aggregate multiple ratings is the majority vote. We first binarize the probability ratings by thresholding them at 0.5, so as to determine whether the peer considered the claim to be more likely to be true or more likely to be false. We consider the claim true if the number of ratings greater than 0.5 exceeds the number smaller than 0.5. We consider the claim false if ratings below 0.5 dominate. (Ratings of exactly 0.5 are not counted, and in case the numbers of votes for and against the claim are equal, we perform a random tie break.) The accuracy of the majority vote approaches 70% when 10 or more ratings per claim are used (Fig. 2). The majority vote is significantly more accurate than trusting a random peer’s rating when $3$ or more ratings per claim are available (Fig. 2, paired one-tailed $t$-test, $p<0.05$, Bonferroni-corrected for $8$ different numbers of ratings per claim). This provides a first simple illustration of the wisdom of the crowd [14, 50].

All statistical comparisons of collective-inference algorithms in this paper rely on a 2-factor bootstrap procedure that treats both peers and claims as random effects. See Methods for statistical procedures and Supplementary Information (section Idiosyncrasies of random ratings, majority vote, and median rating) for discussion of the case of two ratings per claim and of the median rating.

The majority vote binarizes the ratings, which removes information. Extreme ratings closer to 0 or 1 reflect greater confidence than ratings close to 0.5. A simple aggregation rule that gives greater pull to extreme ratings is the rating average. If we average $10$ ratings per claim, the accuracy of our collective inferences increases to about 73%, and averaging $100$ ratings per claim yields about 75% accuracy. The rating average is significantly more accurate than the majority vote when $3$ or more ratings per claim are used (Fig. 2). This provides a first indication that the information about confidence contained in continuous probability ratings is useful for collective inference. (The advantage of using continuous rating information is also evident in the context of inference using judgment-generative models and discriminative supervised models. These results are shown in Fig. 4 and are described below.)

Averaging might be a good approach if each peer judged on the basis of the same evidence. In particular, if each peer used a sampling algorithm for computing a posterior probability for each claim and computed the same number of samples, then the average of the probability ratings would give the posterior for the pool of all samples computed in a distributed fashion by the crowd as a whole.

If the probability ratings do not all reflect the same evidence, then probability averaging is not the optimal way of combining the ratings. Let us consider the opposite scenario where the probability ratings reflect independent evidence, and also assume, for the moment, that the ratings are well-calibrated. A well-calibrated rating is one that accurately reflects the peer’s uncertainty, such that among all claims receiving rating $r$, the rate of true claims is $r$ (so, for example, among claims rated $r=0.8$, 80% are true). If the peers’ ratings are independent given the truth value of a claim and well-calibrated, the optimal aggregation rule is the Independent Opinion Pool [51]: We multiply the binary probability distributions $[r_{i},1-r_{i}]$ across peers indexed by $i$ and renormalize the resulting pair of values to sum to unity (to provide a proper binary probability distribution). Equivalently, we can convert the probability ratings to logits (log odds), sum these, and convert back to a probability as our collective inference (details in Methods). In terms of accuracy, the logit sum (i.e. the independent opinion pool) is equivalent to the logit average. In either case, the sign of the aggregate determines whether the inferred probability is greater than or less than 0.5. The logit average reaches an accuracy of about 77% for 50 or more ratings per claim and is significantly more accurate than the rating average for $3$ or more ratings per claim (Fig. 2). The effectiveness of the independent opinion pool suggests that there is some value in taking the ratings seriously as approximately calibrated indications of probability that are not entirely based on the same evidence.

Summing or averaging the logits yields collective inferences that are equivalent in terms of accuracy. However, if we sum the logits (implementing the independent opinion pool), the collective inferences are highly overconfident: For many ratings, the collective probability will be close to 0 or 1, and will not be well-calibrated. This indicates that the independent opinion pool’s assumption of independent ratings (given the truth of the claim) is incorrect. If all peers instead drew from identical evidence, their ratings would covary given the truth value of the claim. Aggregation of the ratings might still be useful, but only to reduce any noise affecting the ratings. Rating noise might arise at the cognitive and/or motor level, causing random variation of the ratings. The assumption of noisy ratings that reflect the same evidence motivates using the average rather than the sum of the logits. We find that averaging the logits, instead of summing them, makes the collective inference underconfident (Supplementary Fig. 13).

In reality, the evidence on which two people’s ratings of the same claim are based is not expected to be identical or completely independent. Instead, we expect some unknown degree of overlap in the evidence people draw from. For a given claim, there is a limited pool of evidence. Each peer accesses some subset of the relevant facts, and the dependency between peer ratings may reflect the size of the evidence pool and the sources of information available to different peers.

A simple way to account for dependence is to use a convex combination of the logit sum and the logit average, dividing the sum, not by the actual number of ratings, but by an estimate of the effective number of independent ratings. We will return to this issue below in the section Calibrated collective inference requires a reference set of truth-labeled claims.

The independent opinion pool assumes that individual ratings are not only independent, but also well-calibrated. Human probability ratings are known not to be well-calibrated [52, 53, 54, 55]. Consistent with previous findings [6, 56], our participants were overconfident on average. An ideal collective-inference algorithm should correct individual biases in favor of high or low ratings, downweight inaccurate peers, and calibrate overconfident and underconfident peers, so as to optimally combine the ratings.

Bias, inaccuracy, and over- or underconfidence of individual peer judgments can be accounted for by estimating each peer’s calibration function and using the estimate to calibrate the ratings before combining them. Instead of trusting a 0.8 rating to indicate a 0.8 probability of the claim, we can estimate how frequently claims rated around 0.8 by a particular peer are true. For each peer $j$, we need to model the calibration function, which specifies the probability $p_{j}(t=True|r_{j})$ that a claim is true given that peer $j$ has given it rating $r_{j}$. Estimating a peer’s calibration function requires that we have some information about the truth of the claims the peer has rated. If peer $j$ has rated a sufficient number of claims $i$ that we have truth labels $t_{i}\in\{True,False\}$ for, then we can estimate the calibration function. We can then sum or average the logits corresponding to $p_{j}(t=True|r_{j})$ (for the different peers $j$ that have rated a claim) instead of the logits of the original ratings $r_{j}$.

Calibrating the individual peers entails that less informative peers have less influence. A peer whose ratings are unrelated to the truth of the claims will have calibrated logits equal to 0 and thus will not pull collective inference in either direction. More generally, a peer’s calibrated logits will accurately reflect her actual uncertainty.

To estimate the effect of calibrating the logits, we used separate training and test sets of claims for the same set of 376 peers. We designated a random subset of 600 claims as the training set, using these claims’ truth labels to estimate the logistic calibration function $p_{j}(t=True|r)=\operatorname{expit}[\operatorname{logit}[r]/c_{j}-b_{j}]$ for each peer $j$, where $b_{j}$ is the peer’s bias and $c_{j}$ is the peer’s confidence (the factor by which the peer inflates the evidence). We used the ratings by these peers of the other 600 claims as the test set to estimate the accuracy of the individually calibrated logit average (blue in Fig. 2). To simulate performance for different numbers of ratings per claim, we sparsified the data (see Methods). The relatively large number (600) of labeled training claims was chosen to provide an estimate of the potential of individual calibration under ideal conditions.

Averaging individually calibrated logits yielded an accuracy of about 82% when 50 or more ratings per claim were used. The individually calibrated logit average outperformed uncalibrated logit average, with the difference significant whenever 3 or more ratings per claim were used for collective inference ($p<0.05$ for 3, 4, 5, 10, 20, 50, and 100 ratings per claim, Bonferroni-corrected for 8 different numbers of ratings per claim; Fig. 2).

Calibration only improves collective inference if it accounts for individual differences among peers. When we calibrated each peer’s ratings using the calibration parameters estimated for the group as a whole (mid-blue bars in Fig. 2), calibration did not yield an advantage over the uncalibrated logit average. The individually calibrated logit average yielded higher accuracy than the group-calibrated logit average for all tested numbers of ratings per claim ($p<0.05$ for all tested numbers of ratings per claim, Bonferroni-corrected for 8 different numbers of ratings per claim; Fig. 2). These results indicate that collective inference should account for individual differences in judgment behavior.

The individually calibrated logit average requires that each peer has rated a substantial number of claims for which we have truth labels. In many applications, we will not have truth labels at all or not for the claims a particular peer has rated. Ideally, we would like to be able to solve the chicken-and-egg problem of inferring the probabilities of the claims and the propensities of the peers jointly.

The normative approach is to learn a generative model $p_{\psi_{j}}(r|t)$ specifying each peer $j$’s probability density over ratings given the binary truth of a claim. The parameter vector $\psi_{j}$ captures the rating behavior of peer $j$. We find that using a histogram to represent each peer’s truth-conditional rating distribution works well in practice (see Methods for details). We use the expectation-maximization (EM) algorithm to alternately infer the probabilities of the claims and the parameters $\psi_{j}$ capturing each peer $j$’s rating behavior given a true or false claim.

The judgment-generative model achieves an accuracy of about 83% without using any truth labels when 50 or more ratings per claim are used. The judgment-generative model (histogram EM, dark red in Fig. 2) matches the individually calibrated logit average (dark blue in Fig. 2), despite using no truth labels. The inferential comparison revealed no significant differences in accuracy for any of the 8 numbers of ratings per claim (Supplementary Fig. 8). Joint inference of claim probabilities and peer propensities, thus, is a highly attractive approach for collective inference. If we have truth labels for some of the claims, the corresponding probabilities can be set to 0 or 1 in the EM inference, which can further improve collective inference (red line in Fig. 3, right panel).

We saw above that averaging the ratings or their logits outperformed counting binarized ratings (majority vote). An important question is whether continuous ratings also yield collective inferences superior to those based on binary responses when using judgment-generative probabilistic models. Our judgment-generative model predicts a peer-specific truth-conditional probability density over continuous ratings. An influential judgment-generative model that predicts peer-specific truth-conditional probabilities of binary responses has been proposed by Dawid and Skene [57]. Our continuous-response model significantly outperforms this binary response model (Fig. 4, probabilities binarized by thresholding at $0.5$). This finding provides further support for the hypothesis that continuous probability ratings provide better information for collective inference than binary responses.

Inferring claim probabilities and individual behavior jointly with a judgment-generative model for each peer does not require truth labels, but it does require a sufficient number of ratings from each individual. What if we have only a single rating from each peer? In that case, it is not possible to learn a model of each peer’s rating behavior. We could use the logit average in this scenario. However, we might be able to do better than the logit average by supervised learning (using truth labels) of a mapping from a set of ratings to the claim probability.

If we have a training set of ratings of truth-labeled claims, we can use supervised machine learning to predict the probability of a claim from a set of ratings without any modeling of individual peer behavior. One approach that works well is logistic regression on the basis of the centered moments of the ratings. We first compute the mean of the ratings, then center the ratings on this mean. We then compute the mean square (variance), the mean cube (skewness), the mean 4-th power (kurtosis), and the mean 5-th power of the centered ratings. These five numbers characterize the location and shape of the ratings distribution for a claim. A linear logistic regression model takes the five moments as input and assigns a probability to the claim.

When trained with a representative set of peers and claims, centered-moment logistic regression can perform surprisingly well. Trained with a data set of 100 or more ratings of truth-labeled claims, the model performs competitively when given enough ratings per claim for collective inference. Fig. 3 shows how the accuracy of centered-moment logistic regression improves as the training set of truth-labeled claims grows, relative to the other algorithms. Collective inference in these results relies on 188 ratings per claim and performance plateaus at about 83% accuracy. For 75 ratings per claim for collective inference, centered-moment logistic still achieved an average accuracy of about 83% (Supplementary Fig. 14, which shows the dependence on the number of ratings per claim used for inference), significantly higher than the group-calibrated judgment-generative model, the group-calibrated logit-average, and the uncalibrated logit-average ($p<0.05$, 100 bootstrap resamplings of both peers and claims and a randomized train/test split consisting of 100 training claims with known truth value and 1,100 test claims). However, when given 25 ratings per claim or less for collective inference, centered-moment logistic regression was no longer significantly more accurate. Supplementary Fig. 14 shows comprehensive performance results (accuracy, area under the receiver-operating characteristic, and Brier score) for different numbers of truth-labeled training examples (one from each peer of a separate training set of peers) and different numbers of ratings per claim for collective inference. These results show that a sufficiently large set of ratings (with truth labels for at least 100 of the claims) can be useful for collective inference even if we only have one rating per peer and therefore cannot leverage peer-specific models.

Under a range of training set sizes (0, 10, 50, and 300 truth labels), centered-moment logistic regression performs comparably to the judgment-generative model, with the difference in performance not significant (Fig. 4, $p>0.05$ 2-factor-bootstrap paired t-test, df=187). A variant of the logistic regression model using binarized ratings performs significantly worse than the logistic regression model using continuous ratings with a training set of 300 claims (Fig. 4, $p<.05$ 2-factor-bootstrap paired t-test, df=187). This demonstrates the value of continuous probability ratings in the context of supervised models.

The accuracy of collective inferences, which we have focused on thus far, provides one important indicator of collective-inference performance. It can be evaluated for algorithms that produce binary decisions, such as the majority vote, as well as for continuous probability estimates, where it is defined as the rate with which collective inferences fall on the correct side of 0.5. Probabilistic collective inferences, however, should be not only accurate, but also well-calibrated.

We used the truth labels to fit the logistic calibration function (as already introduced above in the context of individual ratings) to the collective inferences $\hat{p}$ of different algorithms: $p(t=True|\hat{p})=\operatorname{expit}[\operatorname{logit}[\hat{p}]/c-b]$. An algorithm is well-calibrated if the calibration function is close to the identity (with bias $b=0$ and confidence $c=1$). Results are shown in Fig. 5 as well as Supplementary Figures 12 and 13. Choosing a random rating exhibits the peers’ general overconfidence. The rating average is underconfident. As reported above, the logit sum (independent opinion pool) is overconfident, whereas the logit average is underconfident. The group- and individually calibrated logit averages are similarly underconfident, reflecting the assumption of entirely dependent ratings. The group and individual judgment-generative models are overconfident, because like the independent opinion pool they assume that ratings are conditionally independent given the truth of the claims.

A straightforward way to calibrate collective inferences is to pass the probability estimates through their calibration function (as we do when we calibrate the ratings of individual peers). This calibration step requires a set of truth-labeled claims. For collective inference algorithms that assume truth-conditional independence, calibration can correct for the overestimation of the evidence that results from the conditional dependency among ratings given the truth of the claims. We used a random subset of 50 training claims and 10 or 20 ratings per claim to calibrate the probability estimates of the algorithms. We then assessed the calibration on an independent test set of 950 different claims (Figure 5, Supplemental Figures 12 and 13). Calibration generalized successfully to the test set: The test-set calibration curves closely track the identity line.

The centered-moment logistic regression model is trained with truth labels using the cross-entropy loss. This amounts to optimizing calibration on the training set. Centered-moment logistic regression, too, exhibited good calibration also on the test set (not shown).

Humans vary in their probability judgment behavior [58, 53, 59, 60, 31]. Beyond interindividual variation of traits and abilities, the behavioral context is known to affect judgments [1, 61, 62, 63, 64, 65]. A particular concern is that motivations created by feedback and by real or imagined rewards might distort ratings and create a challenge for collective inference. In order to assess the robustness of collective inference to variation in feedback and rewards, we randomly assigned each participant to one of seven feedback and reward conditions: In condition 1, no feedback was given. In condition 2, the correct answers were revealed after each block. In conditions 3-7, the correct answer was revealed after each trial. In condition 3, no other information was given. In conditions 4-7, subjects were additionally given the performance of other peers on the presented claim (condition 4) or imaginary coin rewards (conditions 5-7; Figure 1a). The imaginary coin reward was chosen to encourage overconfident (condition 5), well-calibrated (proper scoring rule, condition 6), or underconfident (condition 7) ratings (details in Methods, Table 1). These conditions were intended to enhance the variation of judgment behavior across participants and to enable more realistic and conservative estimates of the performance of collective-inference algorithms. Results presented thus far reflect the robustness of collective inferences to both interindividual variation and variation induced by the seven feedback and reward conditions.

We statistically compared the performance of the logit average and the individually calibrated histogram algorithm (EM) across the seven feedback and reward conditions (Supplementary Figure 7). We found no significant differences in collective-inference performance for any of the performance metrics (accuracy, auROC, Brier score; $p>0.05$, permutation $F$-test comparison of intra- and intergroup variance of performance metrics with 2-factor bootstrap, 10,000 draws). Algorithms accounting for individual peer behavior are expected to be robust not only to trait- and ability-related interindividual variation but also to context-dependent motivational variation (when the context is constant for each peer, as was the case here). For example, the influence of the different coin rewards on peer confidence would be counteracted by algorithms that correct individual miscalibration. However, even the logit average, which does not adapt to individual peer behavior, proved quite robust across feedback conditions.

We also analyzed how individual rating behavior depended on feedback and reward conditions. In particular, we evaluated how feedback and rewards affected confidence, bias, area under the receiver-operating characteristic, Brier score, and accuracy (Supplementary Figure 6). We found no significant overall association between any of the five descriptors of rating behavior and the feedback and reward condition ($p>0.05$, permutation F-test comparing intra- and intergroup variance with 2-factor bootstrap, 10,000 draws). Trait- and ability-related interindividual differences in accuracy and calibration are more pronounced than differences caused by variation across our feedback and reward conditions. The lack of a significant effect of our experimental variation of feedback and rewards suggests that the instruction to judge probabilities invokes a cognitive process that is somewhat insensitive to the manipulations we implemented. Overall these results demonstrate that collective-inference algorithms can work robustly despite substantial variation across individuals in terms of accuracy and calibration as well as the subtler variation caused by context-dependent motivational factors.

Random rating. A baseline estimate that avoids aggregating multiple probability ratings for a given claim is to simply pick a rating at random.

Majority vote. Perhaps the simplest (and most common) method of judgment aggregation is majority vote. We count how many probability ratings are greater than 0.5 and how many are smaller than 0.5 (ignoring judgments that are exactly 0.5). The majority vote estimate is 1 if there are more ratings greater than 0.5 and 0 if there are more ratings smaller than 0.5. If the two counts match, a random tie break is performed. Peers’ confidence is ignored in the majority vote, and all judgments have an equal impact towards the collective estimate. Since the collective inference is binary, it can be evaluated in terms of the accuracy, but not in terms of the auROC or the Brier score.

Rating average. The rating average is the arithmetic average of all ratings of a claim. The confidence of a peer’s probability judgment, thus, has an impact on the collective inference, with more extreme ratings influencing the collective inference more.

Median rating. The median rating is the median of all ratings of a claim. When the majority vote is 1, the median rating will be above 0.5. However, see section Idiosyncrasies of random ratings, majority vote, and median rating in the Supplementary information for some subtleties.

Most accurate peer. Given a training set, we can sort peers according to each peer’s rating accuracy on the training set. The aggregate estimate for a claim in the test set is the rating from the peer who had the highest accuracy on the training set (with random tie-break in case more than one peer achieved the highest accuracy).

Logit sum and average. If we assume that all peers are independent and that the prior probability of a claim being true or false is equal, then the probability that claim $i$ is true ($t_{i}=1$, where t is a vector of size $M$ of binary truth values) given a set of peer probability judgments $\mathbf{r}_{i}=[r_{i1},r_{i2},...,r_{iN}]$ is:

where $\sigma$ is the logistic sigmoid (expit) function. This is also known as the independent opinion pool [51]. If we further assume that peers’ probability judgments are well-calibrated, we have

In reality, ratings are not independent because peers draw from overlapping evidence pools, so this estimate is generally overconfident.

Calibrated logit sum and average. It is known that human probability ratings are not well-calibrated [52, 53, 59, 54]. Given a training set of truth-labeled claims and a peer $j$’s corresponding probability ratings, we can learn the peer’s confidence $c_{j}$ and bias $b_{j}$ using logistic regression:

As before, the logits are summed if we assume each peer is independent. We call this model the individually-calibrated logit sum (or average if we average the logits). In general, we find estimates for these models are more stable if we average the logits. In particular, truth inferences do not necessarily become more extreme in confidence as the number $N$ of ratings from different peers increases.

We may choose instead to learn a global set of logistic regression parameters $c$ and $b$ by combining all peers’ ratings and fitting a logistic regression model. We call these models group-calibrated.

Judgement-generative model learned with expectation maximization. An alternative to variants of the independent opinion pool is to model the truth-conditional rating behavior of peers: $p(r_{ij}|t_{i})$. On this basis, we can model the joint density over all peers’ ratings of claim $i$: $p(\mathbf{r}_{i}|t_{i})$. We assume peers are independent, so the joint density is the product of the individual peer rating densities. We can then use Bayes’ rule to infer the probability of the claim:

In order to learn the peer behavior models $p(r_{ij}|t_{i}),$ we can either use a training set with truth labels or infer them by using the Expectation-Maximization (EM) algorithm [67] to fit peer parameters while estimating the probability of each claim, as is the general strategy in [57]. Note that the EM algorithm can be also used as a form of semi-supervised learning by replacing its inferences for labeled data points with their corresponding truth labels.

The EM algorithm first calculates the posterior claim probabilities of each claim given the current estimate of user/claim traits (Expectation step) and then maximizes the expected value of the joint log-likelihood of the complete data (ratings and truth values) under the previously calculated posterior claim probabilities (Maximization step). Under our model, the EM objective function takes the form:

where $\mathbf{t}$ is the vector of truth values and $\mathbf{R}=[\mathbf{r}_{1},\mathbf{r}_{2},...,\mathbf{r}_{M}]$ is the ratings matrix. If we assume the peers are independent and equal probability for true and false claims, we have

where $J_{i}$ is the set of indices of peers who rated claim $i$ and $\pi_{i}$ is defined as $p(t_{i}=1|r_{ij},\theta^{(\text{old})}).$

Since each peer has her own set of parameters $\theta_{j}$, maximizing $Q(\theta|\theta^{\text{old}})$ splits into a set of subproblems, one for each user. We choose for our generative model $p(r_{ij}|t_{i};\theta_{j})$ a histogram of 5 evenly spaced bins over the unit interval (with similar results with different numbers of bins and uneven bin sizes). Under this model, the M-step has a closed form solution for each peer. Given a partitioning of the unit interval $0=g_{1}<g_{2}<\cdots<g_{B+1}=1$, where $B$ is the number of bins in our histogram, we have

If the bin spacing is uniform, the normalizing constant is $B\sum_{i\in I_{j}}\pi_{i}$. Replacing $\pi_{i^{\prime}}$ with $1-\pi_{i^{\prime}}$ in the above gives the result for the False-conditional histogram generative model.

Given the peer parameters and a group of ratings of a particular claim, we can infer the probability that the claim is true as follows:

If the users’ ratings are independent, we have:

The inference, thus, involves summing estimates of $\log[p(r_{ij}|t_{i}=1;\theta_{j})/p(r_{ij}|t_{i}=0;\theta_{j})]$ across peers, where $p(r_{ij}|t_{i}=1;\theta_{j})$ and $p(r_{ij}|t_{i}=0;\theta_{j})$ are provided by our generative model with peer-specific parameters $\theta_{j}$. Because of the assumption of truth-conditionally independent ratings, the collective inferences are expected to be overconfident. To avoid modeling the truth-conditional rating dependencies, we propose to take a supervised recalibration approach, which requires a small number of truth-labeled claims (e.g. 100). Using the truth labels, we can recalibrate our collective-inference logit estimates using the same approach as on the level of individual peer ratings: by fitting a bias and confidence parameter, which amounts to training a logistic regression model.

Dawid-Skene. The rating data is binarized by applying a Heaviside function $H(\cdot)$ after subtracting 0.5 from each rating. This maps ratings larger than 0.5 to 1 and ratings smaller than 0.5 to 0. We then fit the two-coin Dawid Skene model [57]. The Dawid-Skene model defines the probability that each user gives the correct (binarized) rating to a claim: $p(H(r_{ij}-.5)=1|t=1)=\theta^{(1)}_{j}$ and $p(H(r_{ij}-.5)=0|t=0)=\theta^{(0)}_{j}$. As with the histogram generative model, we set the prior on $t$ to be flat and we use EM to estimate $\theta_{j}^{(1)}$ and $\theta_{j}^{(0)}.$ Again, the M-step has a closed form solution:

where $I_{j}$ is the indices of the claims that peer $j$ has rated.

Supervised centered-moment logistic regression. Ideally, we would be able to learn a function that maps from the distribution of peer ratings of a particular claim to an estimate of the probability of the claim. The distribution of ratings a claim has received can be characterized by its centered moments. This approach does not require a large number of ratings. The $m$-th centered empirical moment of the set of ratings for claim $i$ is:

where $|\cdot|$ is the cardinality operator and $\mu_{i,J_{i}}:=\frac{1}{|J_{i}|}\sum_{j\in J_{i}}r_{i,j}.$

We characterize the distribution of ratings by 5 real numbers: The first raw moment (mean), the second centered moment (variance), the third centered moment (skew), the fourth centered moment (kurtosis), and the fifth centered moment. The centered-moment logistic regression model fits a weight to each of the five moments of the empirical distribution of ratings in a labeled training set. The weighted combination is passed through the standard logistic function to provide the probability estimate $p(t_{i}=1|r_{i,J_{i}}).$ This model can also be trained on binarized ratings.

We created a set of 1,200 general knowledge claims equally partitioned into six categories, giving 200 claims per category: history, geography, science, social sciences and politics, sports and leisure, and arts and entertainment. Each category had an equal number of true and false claims. Our full list of claims and truth values is provided in the data repository as detailed in the Data availability statement.

Recognizing that the baseline knowledge assumed in this study centered predominantly on U.S. contexts, we sought participants through www.prolific.org who self-identified as U.S. citizens. We further restricted the participant pool to those who had no rejections from previous studies and at least 20 studies completed. Prolific users who met these criteria could begin participation in the study independently and were included on a first-come, first-served basis. The 1,200 claims were augmented by 10 trivially easy claims (e.g. “All fish can fly”) to gauge user engagement. We excluded participants who failed on these claims. Each participant was asked to judge all 1,200 claims across six sessions on different days. Of 504 paid participants, 376 completed all six sessions and only these were retained in the data set. Given the selection process, our sample is thus not representative of either the human population or the U.S. population. However, the remaining group of 376 participants was diverse in age (18 to 65 years, median: 25 years) and gender (61% female, 5% nonbinary, 34% male), and to a lesser extent in race (7% Asian, 6% Black, 1% Native American, 1% Pacific Islander, 82% White; 3% Prefer not to say).

We used www.meadows-research.com as the platform to host our studies. Participants were instructed on how to rate claims with a few trial examples provided before the study began. Each user completed 6 studies, each consisting of a random portion of 200 of the 1,200 claims. The 200 claims were further split into 4 blocks of 50 claims. Users were allowed to begin each study at their own pace. Once a study began, each claim had to be completed within 20 seconds. Each of the included 376 participants judged each of the 1,200 claims.

In condition 1, participants received no feedback or reward. In all other conditions, participants received True/False feedback. In conditions 5, 6, and 7, participants additionally received imaginary coin rewards for accurate judgments. The reward fell off with the discrepancy $|r-t|$ between the rating $r\in[0,1]$ and the truth $t\in\{0,1\}$. Each of conditions 5, 6, and 7 used a different reward function, encouraging overconfident rating (reward $\propto|r-t|$), well-calibrated rating (reward $\propto|r-t|^{2}$, proper scoring rule), and underconfident rating (reward $\propto|r-t|^{3}$), respectively. These conditions were included to enable us to gauge the robustness of the collective-inference algorithms to varying incentives that might distort the human judgments.

The performance measures we report are from a held out test set with the trainable models trained on a disjoint training set. For each bootstrap resample of the data, we randomly partition the data into $K$ equal bins and leave each bin out as a test set and train on the other $K-1$ bins, giving $K$ performance metrics, which we average together.

In real-world applications, we will not have a dense matrix of probability ratings as acquired in our online behavioral experiment. For peers in a social network rating claims they encounter, for example, we expect that every claim is rated by a small subset of the peers, and that every peer rates a small subset of the claims. We therefore create sparse rating matrices by resampling, so as to compare the performance of different collective-inference algorithms as a function of the number of ratings per claim and the number of ratings per peer.

We use a resampling method that we call “thinning” to obtain sparse ratings matrices as may be encountered in practice. Thinning the matrix by factor $k$ along the peers dimension involves replacing each peer’s column by $k$ copies of that column. For each peer and row, we then retain only one of the $k$ identical ratings, choosing the one to retain at random. The other copies are set to “missing”. The new matrix contains the same ratings as the original matrix, but the ratings appear to originate from $k$ times as many peers and only a fraction of $1/k$ of all possible ratings is present. Although the sparsified matrix contains the same set of ratings, it provides less information to a collective-inference algorithm because it does not specify which of the ratings in different columns actually came from the same peer. An algorithm like histogram-EM must try to infer more peer models given fewer ratings per peer.

The same thinning technique is applied to the claims, replacing each claim’s row with $l$ copies of that row. For each present original rating, we retain only one of the $l$ identical copies (choosing which to retain at random). Expanding a dense $M$ by $N$ matrix into a sparse $M\cdot l$ by $N\cdot k$ matrix preserves all ratings, while simulating a larger number of peers ($N\cdot k$) and claims ($M\cdot l$). Subsampling further enables us to control the number of ratings per claim and peer.

Estimates of the accuracy, area under the receiver operating characteristic (auROC) and Brier score of different collective-inference algorithms are affected by measurement noise as well as variation due to the sample of participants and claims. We are interested in statistical inferences that hold, not just for our sample of participants and our 1,200 claims, but for the underlying populations that our participants and claims can be considered random samples from (e.g. U.S. citizen prolific users; see section Behavioral experiment, above, for limitations of our samples of participants and claims). This motivates a conservative approach to frequentist statistical inference in which we treat both participants and claims as random effects. For statistical comparisons among inference methods, we therefore simulate the variation due to sampling of claims and participants by resampling both participants and claims with replacement (two-factor bootstrap) [47].

The two-factor bootstrap provides variance estimates for each performance measure (accuracy, auROC, Brier score) for each collective-inference algorithm as well as variance estimates for the difference in these performance measures for each pair of collective-inference algorithms. For each performance measure and pair of collective-inference algorithms, we compute a performance difference for each two-factor bootstrap sample. The variance of these differences forms the basis for inference using paired $t$-tests. The number of degrees of freedom is set to the smaller of the two numbers of samples (number of participants or number of claims) minus 1.

In Fig. 2, we are interested in the degree to which each step in a progression from naive to sophisticated collective-inference algorithms improves performance as a function of the number of ratings per claim. We therefore perform single-tailed $t$-tests, testing for a performance improvement for each step. We use a Bonferroni correction to account for $9$ comparisons (for different numbers of ratings per claim). In Fig. 4, we use a paired t-test on the performance difference between two models using a two-factor bootstrap to estimate variances. We again use a Bonferroni correction to account for the $4$ statistical tests done for each algorithm comparison across the four panels, corresponding to different numbers of truth labels.

The data set of 451,200 probability ratings (of 1,200 claims by each of the 376 peers), the 1,200 claims (falling in 6 topic categories) and their truth labels will be shared with the community in an open-science repository upon journal publication of the paper.

Our code repository is available to reviewers now and will be shared on GitHub upon journal publication.

Median beats majority vote for even numbers of ratings. For odd numbers of ratings, the median rating is above 0.5 if and only if most ratings are above 0.5. The binary decisions rendered by the median rating and the majority vote will therefore be identical and their accuracy will match exactly. For even numbers of ratings, however, it can happen that the vote is equally split with half the ratings above and half below 0.5. In our definition of the majority vote, a random tie break is then performed. The median rating, by contrast, will average the two central ratings straddling 0.5. The median rating will fall on the side of 0.5 of the more confident one (the one farther from 0.5) of the two central ratings. It can, thus, take advantage of the continuous confidence information in the central two ratings in this rare scenario. The median rating therefore has slightly greater accuracy than the majority vote for even numbers of ratings, whereas it matches the majority vote in accuracy for odd numbers of ratings (Fig. 2).

For two ratings, the majority vote has the same accuracy as a random peer’s rating. The majority vote cannot benefit from the wisdom of a crowd of two people. It yields the same accuracy as picking a peer at random and trusting that single rating (left panel in Fig. 2). In case both ratings fall on the same side of 0.5, majority vote and a random one of the two ratings lead to the same decision, so the expected accuracy is the same for both methods in this scenario. The alternative scenario, where one of the two ratings is above and the other below 0.5, entails a coin flip in majority vote. Choosing a random rating and using a coin flip to break the tie in the majority vote both yield chance performance. The overall accuracy of the two methods is therefore identical.

Given a set of probability ratings $\mathbf{r}=(r_{1},...,r_{N})$ and assuming independent rating distributions conditional on the claim truth value as well as a flat truth prior, we have

If we assume the raters are well-calibrated, i.e., $p(t=1|r_{j})=r_{j},$ then the lhs can be simplified to $\prod_{j=1}^{N}r_{j}$ for $t=1$ and $\prod_{j=1}^{N}(1-r_{j})$ for $t=0$.