When Does Uncertainty Matter?: Understanding the Impact of Predictive Uncertainty in ML Assisted Decision Making

In this project, we seek to understand how people make decisions when provided with the uncertainty of an ML model as an input to their decision process. The uncertainty of an ML model is auxiliary information provided by many ML models that can be used as an input to human decision making in addition to the model’s prediction. Specifically, we explore how different factors related to the model’s uncertainty distribution impact ML assisted decision making. We measure this through user agreement with the model’s prediction before and after being shown the uncertainty information, which allows us to isolate the important factor of agreement with the machine prediction from whether the prediction is correct. We also study how ML assisted decision making differs depending on user expertise in the domain or in ML. We describe each of these factors in more detail below.

We operationalize the notion of predictive uncertainty in a Bayesian setting where predictions may be given as point estimates or as entire posterior predictive distributions. We show uncertainty information as a visualization of the model’s full posterior predictive distribution, in addition to its prediction marked at the mean of the posterior predictive.

We study the effect of 2 factors related to the uncertainty distribution: the shape of the uncertainty distribution, and its variance. Different shapes of distributions have different properties that may both affect how much people are willing to take them into account when making decisions, and what the optimal way to take them into account may be. We study 3 shapes of posterior predictives: a normal distribution, a skewed distribution, and a bimodal distribution. A normal distribution is commonly studied in past work (e.g., Fernandes et al. (2018)), and has the property that its mean is equal to its mode. This is not true however for the bimodal distribution, which has 2 modes and very little probability near its mean, or the skewed distribution which has a mode to one side of the mean and a long tail of reasonably high probability outcomes. Whether people will still update their estimates to reflect the ML model’s prediction when presented with these distributions with more complicated properties is an open question.

We also explore the effect of variance on people’s behavior. We wish to answer the question of how much uncertainty affects people’s decisions when the variance of the posterior distribution is larger vs. when it is smaller. Are people more likely to agree with an ML prediction when the uncertainty around the prediction is low? That is, do they correctly interpret and respond to uncertainty? And how does this compare to when only shown a point estimate? We study this in the context of the normal distribution.

Moreover, we explore how decision differs between experts (in either the domain or in ML) and non-experts. Will people with differing levels of ML expertise feel more or less compelled to follow the machine’s prediction regardless of the provided uncertainty? Will they respond more or less strongly to the presented uncertainty estimates? Orthogonally, will people who know more about the domain we study feel more confident in their prediction and therefore place comparatively less weight on the machine’s prediction? We study the relationship between these notions of expertise and decision making with various forms of uncertainty.

Finally, we study the correlation between subjective measures of trust and perceived usefulness of the ML model and participant agreement with the model. In summary, our main research questions are:

• •

  Does showing predictive uncertainty affect how closely people follow model predictions?

• •

  Does the effect of showing predictive uncertainty depend on the type of uncertainty – either the shape or the variance of the distribution?

• •

  Do participant with more expertise, either in the domain or in ML, more closely follow model predictions?

In this section, we describe the details of the user study that we carried out to answer the research questions outlined in the previous section.

We look at 3 metrics to understand the extent to which seeing predictive uncertainty affects people’s decisions: the distance (in the $L_{1}$ norm) between the first estimate (before seeing any model outputs) and the second estimate (after seeing the prediction and uncertainty wherever applicable) – we call this the “update”; the distance between the second estimate (after seeing the model’s prediction) and the model’s prediction – we call this the “final disagreement”; and the distance between the first estimate (before seeing the model’s prediction) and the model’s prediction – we call this the “initial disagreement”. The update allows us to determine the extent to which participants updated their estimate based on the model’s prediction and uncertainty information. The final disagreement measures the extent to which people agree with what the model predicted. The initial disagreement allows us to understand any baseline differences between participants of varying levels of expertise before they were shown the model’s prediction and uncertainty information.

Our primary analyses use mixed-effects regression models for these three metrics. Unlike regression models with only fixed effects, mixed effects models allow one to properly account for correlation between repeated measurements from participants in the study (e.g., some participants may consistently change their estimates a large amount while others may consistently change their estimates by a small amount) (Bates et al., 2015). A random intercept was included for the participant ID in all models. No other random effects were included in the models. All the relevant factors including the type of predictive uncertainty, background in ML, experience living in Cambridge, self-reported usefulness and self-reported trust in the model were included as fixed effects in their respective analyses. All p-values reported are derived from a one-way ANOVA based on these models. We report p-values that are less than $0.05$ as significant and p-values in the interval $[0.05,0.10]$ as marginal. The bar plots illustrate the sample mean of the outcome in the relevant group of participants $\pm 1$ standard error based on the fitted model.

Our first set of research questions are about how people’s willingness to update their estimates differed based on their familiarity with either the domain (apartment rentals in Cambridge, MA) or ML models. Table 1 captures the number of responses (i.e., the total number of apartment pricing examples completed) and participants from each category of ML expertise and domain expertise. Figure 3 illustrates the magnitude of the update (left), final disagreement (middle), and initial disagreement (right) among participants in each of the two categories of expertise in Cambridge apartment prices (top) and two categories of ML expertise (bottom). Throughout, we combined the ML expertise categories of “No understanding or little understanding” and “Worked with it a few times” into a single category which we refer to as “No ML background / Some ML background”, and we relabelled the category “Worked with it many times or on a larger project” to “Strong ML background” for clarity.

We found that participants with domain expertise in Cambridge apartment prices had lower initial disagreement and smaller updates compared to those without such expertise. However, the final disagreements of both groups were similar. Participants who have previously lived in or around Cambridge made initial estimates that were closer to the model’s prediction (smaller initial disagreement) even before having seen the prediction compared to those who never lived in Cambridge (significant: $F(1,188)=14.4761$, $p=0.0002$). This likely reflects their increased familiarity of rental prices in Cambridge. However, we also observed that domain experts had smaller updates (significant: $F(1,188)=23.0224$, $p<0.0001$), and there was no significant difference between domain experts and non-experts in their final disagreement with the model ($F(1,188)=0.8631$, $p=0.3541$). This suggests that the ML model equalized differences in the initial disagreement between experts and non-experts by bringing participants’ final disagreement with the model into the same range, regardless of expertise.

We found that participants with more ML expertise updated their estimates less after seeing the model’s prediction (significant: $F(1,188)=5.1232$, $p=0.0248$). Similar to the analysis comparing domain experts in Cambridge apartment prices to non-experts, (i) the final disagreement did not significantly differ between the ML experts and non-experts, and (ii) the ML non-experts had smaller initial disagreement (marginally significant: $F(1,188)=3.1833$, $p=0.0760$). These trends may be attributed to the ML expert group having a greater portion of participants with domain expertise in Cambridge compared to the ML non-expert group. When accounting for domain expertise (i.e., by including a fixed effect term for domain expertise in our mixed-effect regression models), there were no significant differences between ML experts and non-experts in their initial disagreement, magnitude of update, or final disagreement.

Our second set of research questions are about the extent to which showing uncertainty influences the decisions people make, and how the type of uncertainty modulates that effect. To this end, we plot the magnitude of the update (left), final disagreement (middle), and initial disagreement (right) for each of the five conditions corresponding to a different kind of uncertainty or no uncertainty (Figure 4).

We found that the type of uncertainty affected participants’ final disagreement with the model. Furthermore, showing no uncertainty resulted in final estimates that were farthest from model predictions. The type of uncertainty had an effect on how closely participants’ second estimate agreed with the machine’s prediction (size of the final disagreement) (marginally significant: $F(4,185)=2.2149$, $p=0.0690$). It can be seen from Figure 4 that the condition where participants were shown a normal distribution with low variance moved people closest to the model’s prediction, however even in cases like the normal distribution with high variance where the model is clearly uncertain, or the bimodal distribution where there is very little probability mass on the model’s prediction, people were still moving their second estimates closer to the model’s prediction compared to when they were shown no uncertainty. More specifically, participants shown uncertainty had an average final disagreement $49 smaller than those not shown uncertainty (significant: $F(1,188)=8.3989$, $p=0.0042$).

We do not see statistically significant differences in the magnitude of the update ($F(4,185)=1.5239$, $p=0.1969$). This could be tied to high variance in the original estimate before being shown the prediction, although these are also not significantly different ($F(4,185)=1.2734$, $p=0.2820$), suggesting that our randomization worked.

Finally, we present an analysis of the impact of uncertainty type stratified by participant expertise. The impact of the type of uncertainty on agreement with the prediction may be modulated by expertise. Table 2 shows the number of responses and participants in each uncertainty group, stratified by different levels of domain and ML expertise. While the sample sizes are small to draw strong conclusions, this analysis helps us gain further context on how expertise and uncertainty type interact, and suggests areas for future research. Figure 5 illustrates the magnitude of the update (left), final disagreement (middle), and initial disagreement (right) for each uncertainty type/no uncertainty among participants in each of the two categories of domain expertise. Figure 6 shows analogous results with respect to ML expertise.

Uncertainty type by domain expertise: While we found that the magnitude of the update and initial disagreement were much higher for people without domain expertise, the relative trends of the impact of the type of uncertainty were similar for domain experts and non-experts. The normal low-variance and skew conditions had amongst the largest updates for both domain experts and non-experts, and the normal high variance condition had the smallest updates for both groups. For both domain experts and non-experts, the initial disagreement was similar between the different uncertainty conditions, as one would expect due to the randomization.

Uncertainty type by domain expertise: We found that participants in the no uncertainty condition had the largest final disagreement for both domain expertise groups. The final difference between the participants’ estimates and the model’s predictions were relatively consistent between the different types of uncertainty for both groups. For domain non-experts, the final disagreement was largest in the no uncertainty condition. While this was also true for domain experts, it was not as marked.

Uncertainty type by ML expertise: We found that ML experts updated their original estimates more with the normal low variance condition than for any other, while there were no clear trends for ML non-experts. From the left column of Figure 6, the normal low variance condition had much larger updates for experts (top) than those with only some ML experience or no ML experience (bottom). For the experts, the updates were much larger for this condition than for any of the others, while the updates were somewhat similar for all of the other uncertainty conditions and the no uncertainty condition. Perhaps ML experts are used to working with normal distributions and trust the prediction when the ML model is relatively certain, but do not agree to the same extent ML models that do not present their uncertainty, or present uncertainty estimates that have high variance and possibly hard-to-interpret distributions. For the non-experts, the trends were not as clear, and the updates appear relatively similar for the different uncertainty conditions. Perhaps participants with less experience with ML do not consider the variance of the distribution as much and have less of a prior on which uncertainty displays from the model are reliable. Further exploring this hypothesis is interesting future work, as it suggests that ML non-experts are perhaps less likely to respond appropriately to uncertainty.

While our study makes one of the first attempts at investigating the impact of different types of posterior predictive distributions in ML assisted decision making, it has limitations which may affect the generalizability of our findings. First, while the domain experts in our study had significantly more knowledge of apartment prices in Cambridge compared to the non-experts as evidenced by having significantly smaller initial disagreement with the model, it would be a good idea to consider users with a higher degree of domain expertise such as local real estate agents. In this case, we would expect greater differences in decision making between domain experts and non-experts. Second, as our study focuses entirely on the task of estimating apartment prices, results may differ for other decision making tasks and in other contexts. For example, human behavior studies have found that participant behavior can vary considerably between different tasks or in different contexts due to factors such as risk sensitivity (Lakkaraju & Bastani, 2020; Poursabzi-Sangdeh et al., 2021; Attali et al., 2011). Third, the two-stage nature of our study design (i.e., participants providing price estimates prior to and after seeing the model output) may be susceptible to self-priming effects (i.e., participants’ price estimates after seeing the model output may be affected by having to provide initial price estimates without seeing the model output). To help mitigate potential self-priming effects, we offered a $30 incentive for users to make the most accurate apartment price predictions. Furthermore, this study design has been suggested as good practice by prior work (Poursabzi-Sangdeh et al., 2021).

Besides the type of posterior predictive distribution and expertise of the decision makers, there may be other important factors that affect the impact of predictive uncertainty in ML assisted decision making. One such factor may be model performance. For example, if decision makers are aware that the model has excellent performance, they may heavily rely on the model regardless of their expertise or the shape or variance of the posterior predictive distribution. Exploring the impact of model performance in this context would be an interesting avenue for further work.

The setting we considered is ML assisted decision making under limited information. In this context, one can consider scenarios where users have less than, more than, or the same features as the ML model. Each of these three scenarios has been studied by seminal works in the ML assisted decision making literature. For example, Poursabzi-Sangdeh et al. (2021) studied scenarios where users have the same features as the ML model and scenarios where users have more features than the ML model. As one of the first studies investigating how different types of posterior predictive distributions affect ML assisted decision making (and how such effects vary between domain/ML experts and non-experts), we did not want to introduce additional complications arising in the scenario where users have fewer features than the ML model. For instance, if the ML model includes some additional features (not available to the user) that are highly influential for a few apartments, model predictions may diverge heavily from experts’ understanding and create distrust in the model. Exploring these additional scenarios would be interesting future directions.

Finally, note that there are several sources of predictive uncertainty, which have been categorized into data uncertainty (also referred to as aleatoric uncertainty), model uncertainty (also referred to as epistemic uncertainty), and distributional uncertainty (Malinin & Gales, 2018). Prior works have highlighted the importance of these different sources of uncertainty (e.g., Dusenberry et al. (2020) on model uncertainty). While our work focused on posterior predictive distributions – which captures data and model uncertainty (Dusenberry et al., 2020) – it would be interesting to investigate the impact of presenting all these three of these types of uncertainty to decision makers.

We would like to thank the reviewers and action editor for their comments which helped us improve the paper. SM and IL were supported by the National Science Foundation Graduate Research Fellowship Program under Grant No. DGE1745303. SM was also supported by the National Library Of Medicine of the National Institutes of Health under Award Number T32LM012411 and Fonds de recherche du Québec – Nature et technologies B1X research scholarship. HL is supported in part by NSF awards #IIS-2008461 and #IIS-2040989, and research awards from Google, JP Morgan, Amazon, Bayer, Harvard Data Science Initiative, and D^3 Institute at Harvard. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the funding agencies.