Communicating Uncertainty and Risk
in Air Quality Maps

Publicly available sources of air quality data exist worldwide. Many governments operate air quality sensors and, to varying degrees, make their data available online. In the United States, the Environmental Protection Agency operates AirNow, a site showing a contour map colored by estimated air quality category (Figure 1(a)). This estimate is an interpolation of the data from air quality sensors across the country. Colors indicate categories of health risk, each with corresponding guidelines (see Fig. 3).

Recently, low-cost, internet-connected air quality sensors have become popular, such as those from PurpleAir. These sensors are installed by individuals in and around homes and buildings; their data are available online in real time. Low-cost sensors, offering better availability and wider spatial coverage of air quality data, could have a transformative effect on the public’s awareness [9]. Websites like IQAir and PurpleAir show estimates of air quality from these sensors. These visualizations may show contours, indicate sensor values directly as in Figure 1(b), and/or show glyphs with aggregated sensor data. The interface gives a number summarizing the current pollution level in that location.

In general, publicly available sources for air quality information do not include uncertainty. By neglecting uncertainty, predictions shown on sites like AirNow may often be inaccurate or misleading, and may underestimate air pollution in general. The raw data from sensors and the interpolation used to create a map are both sources of uncertainty, but users typically see no indication of either.

The placement of sensors is a primary source of uncertainty. For example, sensors are not evenly spaced across the U.S., and their distribution does not reflect the population distribution or the most significant pollution sources. For example, the interpolations shown on AirNow often do not capture variations on the scale at which pollution from traffic on highways is present, and sensors are often placed away from the population centers of cities. While government-owned sensors are helpful for monitoring long-term, large-scale air quality trends, they may not be offering completely truthful information in places where pollution is a chronic risk to public health. Air pollution from wildfires is also increasing drastically; many heavily affected areas are sparsely measured by the government’s network of air quality sensors. Air quality estimates that users see, then, often do not reflect the most relevant sources of pollution and the potential variability in air quality.

Another source of uncertainty in air quality maps is the algorithm used to convert air sensor readings into a contour map. The contours shown on AirNow are based on Inverse Distance Weighting (IDW), a deterministic algorithm that predicts the value at a location by weighting detections from nearby sensors. IDW is widely used, but its output is highly variable depending on the specific parameters used (see Figure 2). The parameters used for the AirNow site differ significantly from parameters found to be optimal in other studies of air pollution interpolation [10]. Techniques like cross-validation can be used to optimize the parameter values, but this is less effective when interpolating over large spaces, such as the vast distances between sensors in some parts of the United States. More sophisticated approaches exist (kriging is the most prominent alternative) but any approach requires assumptions and is prone to error.

Though contour maps created from interpolations are widely used in geospatial communication, their uncertainty properties are not well-studied [11]. Uncertainty in how sensors are labeled, or artifacts from their binning, have a significant impact on the resulting contours. Researchers have proposed a method for identifying areas in choropleth maps where labeling has a strong effect on visual boundaries [12]. With this information, a map designer could adjust labeling to reduce potential bias. In the air quality case, guidelines exist for ranges of Air Quality Index (AQI) values (see Figure 3), so a standard binning is already established. Maps need to be updated with evolving air quality readings, so manually adjusting maps is not feasible.

Research suggests that uncertainty information may help people make better decisions in their daily lives. For example, mobile transit apps showing uncertainty in bus arrival times may help people optimize when to leave for the bus [13]. Including numeric uncertainty in weather forecasts may increase trust and help people make more holistic decisions [14], and the general public understands and expects uncertainty information in weather forecasts [15].

For risk communication in particular, research suggests that well-designed visual aids can be transparent, effective tools for un-biasing people’s perception of danger [16]. Some recent human-computer interaction work has focused on informing the public about underreported risks they face from flooding  [17]. Uncertainty visualization enhances risk communication: users may be more willing to take appropriate precautions when shown an uncertain weather forecast compared to a categorical warning [18]. If an uncertainty-aware map shows the competing claims from air quality data sources, people may be able to make more informed decisions and understand the sources and characteristics of air pollution in their city compared to others.

One challenge in characterizing people’s understanding of air quality information is that many people in the United States may not regularly engage with it. One study found that about 12% of the population had changed their behavior within the past year in response to poor air quality, but those with respiratory conditions are more likely to take action [19]. We conducted an online survey to gauge people’s awareness of local air quality, their sources of information, and their responses to poor air quality.

We decided to focus on users who already have some awareness of air quality, in order to yield respondents who are motivated to pay attention to their health and may have taken preventative action in the past. We targeted people 18 and older in our local community, which had experienced significant air pollution from wildfires within the past year. We had 54 respondents, 40 of whom fully completed the survey. Of the respondents who indicated being aware of poor air quality within the past year ($n=34$), $91\%$ specified that this unhealthy air was due to a wildfire.

In general, our survey indicated that people aware of air quality issues are interested in simple AQI information and are likely to make changes and trust sources of information, but may not have a consistent rationale or source for these decisions. When asked whether they feel they have accurate, complete information about the air quality near them, $72\%$ ($n=39$) at least somewhat agreed.

We asked respondents whether they had used any websites in the past 12 months to check air quality; $78\%$ indicated they had. Of those who did, $35\%$ used AirNow alone, $32\%$ used another site alone, and $32\%$ used a combination of AirNow and another site. At least $67\%$ of respondents, then, use a contour-based air quality map.

Most respondents ($91\%$, $n=34$) indicated that they had changed their behavior within the past year due to poor air quality. The changes these respondents have made are often significant: $90.3\%$ of those people said they had avoided time outside, $74\%$ cancelled or skipped activities, and $65\%$ did less strenuous exercise.

In this study, we focus on the harmful pollutant PM2.5, which comes from automobiles and wildfires, among other sources. AirNow is considered the ground truth in the U.S. for air pollution readings, but the sensor coverage is limited in many areas of the United States, and in more densely populated states, sensor coverage is disparate from the distribution of people and sources of pollution (e.g. roads).

PurpleAir is one of the most prominent sources for low-cost air quality sensors. Individuals around the world can purchase sensors to install inside or outside homes or other buildings (we consider only outdoors sensors here). These sensors are connected to the internet and their data are made available at purpleair.com in real time. Because PurpleAir sensors are not maintained by the company once they are installed by individuals, their accuracy may worsen over time. In particular, dust and other debris accumulates over the laser-based sensors, and without cleaning, the readings may drift. These sensors do not directly measure PM2.5 concentration, instead inferring it from other measurements.

One source of uncertainty is the interpolation approach, as described in section 1.1.1. Two common approaches in geospatial applications are inverse distance weighting (IDW), which is deterministic, and kriging, which is probabilistic. IDW, used for AirNow, predicts values at points by taking a weighted average of the $k$ nearest neighbors. The results of IDW are highly dependent on the parameters used, $k$ and $p$, where the $k$ nearest sensors are weighted by their distance to a location, $\frac{1}{(\mathrm{dist}^{p})}$.

Kriging is a statistical approach based on characterizing the autocorrelation between pairs of detections. This approach requires more tuning than IDW, such as specifying the shape of the semivariogram describing the autocorrelation. Previous work examines the use of kriging vs. IDW for interpolating air quality, finding that which method is superior depends highly on the scenario and on the particular pollutant [10]. In this study, we use kriging to generate visualizations because of its more natural relationship with uncertainty; kriging algorithms estimate a mean value and standard deviation at each grid point.

Interpolation only (Figure 4) shows no uncertainty, representing the status quo in air quality visualization. Contours are based on the mean interpolated kriging estimate from sensor detections (see Section 4.2). We include this view as a baseline to understand typical reasoning.

Interpolation with sensors (Figure 5) represents a standard type of visualization available on air quality monitoring sites. (Often, these views only show sensors and do not include interpolation; we left that case out of this study.) Research suggests that users prefer to be able to view conflicting individual data sources even when an aggregation—in this case, the contour map—is available [23]. We indicate relative reliability by encoding the government-owned sensors with larger circles.

This view lets us ask how people’s understanding of an interpolation changes if the underlying information is also shown. Discrepancies between the raw data and the chosen interpolation are an implicit representation of uncertainty. Conflicts between different sensors show users some ambiguity, perhaps encouraging thought about how the interpolation was derived from the data, especially when the measurements seem disparate from the interpolation.

Prior work suggests how users might interpret these maps: people may aggregate information differently if given access to uncertainty information, weighing each source of information and taking its reliability into account [22]. Specifically, users are likely to mentally average the sensor information together to reach a conclusion, maybe using a weighted average if the sensors have different reliability.

Researchers have proposed thinking of a set of possible outcomes in an uncertain situation as multivalued data [24]. Distinct from multivariate or multidimensional data, in multivalued data, each datum has a collection of values for a single variable (in our case, possible AQI values at each location). The authors pointed out that few geospatial visualizations had treated uncertainty as multivalued data without using animation. Multiple linked displays have been used for exploring ensembles of outcomes of simulated geospatial data [25]. In the interest of public accessibility and distribution potential, however, we limit ourselves to static views. Integrating uncertainty into the map itself is likely to be more influential for people’s decisions than providing an adjacent uncertainty view, and it may be easy to ignore uncertainty information presented separately [26].

Significant work in geospatial visualization has focused on techniques that can be integrated into static views, including textures, transparency, hue, and value [27]. For example, bivariate color maps have been proposed to integrate data and uncertainty into one image [28]. A variant of this idea is the Value-Suppressing Uncertainty Palette (VSUP); encoding information and uncertainty in a VSUP encourages reasoned, uncertainty-aware decision making [29].

In general, these approaches are more suited to showing the magnitude of uncertainty rather than depicting the relative probabilities of possible outcomes. Texture and color-based approaches work by downplaying or obfuscating more uncertain information. Their underlying premise is that differences among more certain data are more important than differences among highly uncertain data [29].

Another approach to uncertainty visualization is to fairly represent the relative likelihoods of different outcomes. Recent work in uncertainty visualization has shown promise in direct displays of these ensembles of potential outcomes. For example, showing a sampled ensemble of hurricane predictions can improve users’ ability to estimate danger over summary displays, which depict the mean and its spread [30], [31]. In this study, we focus on transferring direct displays into a map context. We considered a wide range of designs that could show direct displays of uncertainty, ultimately using the following four designs.

To quantify the uncertainty information underlying these views, we use kriging to describe a range of possible outcomes from measured sensor data. Figure 6 shows how we sample this kriging grid to convey uncertainty for different visual designs. The mean kriging-based estimate for each grid cell is used to create the contours in the standard views in Section 4.1. For the interpolation only option—showing a contour map without any uncertainty—the interpolated visualization has a higher resolution, i.e., it samples more points on the kriging grid. When we add in uncertainty information, we sacrifice some of the space available for showing the mean estimate in exchange for more information about the standard deviation.

Small multiples (Figure 7) align with current research in uncertainty visualization. They are a frequency-framing way to understand the uncertainty inherent in an estimate, showing the different possibilities all at once. Showing uncertainty with discretized outcomes may improve user recall [32] and help with confident, optimal decisions [13].

In general, small multiples are not used to convey discretized uncertainty, though encoding comparisons of “multiple realizations” in geospatial uncertainty visualization has been proposed [33]. In our case, it is a way to show uncertainty via a direct display of possible outcomes. We propose that this view might help users make optimization judgments like whether to reduce their outdoor activity.

Previous work in visualizing multiple kriging results of air quality data suggests that interactivity is vital, for users to see how the probabilities change according to threshold [34]. Without interactivity at our disposal, small multiples capture representative snapshots reflecting this type of reasoning, showing the map at each of nine thresholds. Note that in our case, the small multiples can be ordered from most optimistic to most pessimistic scenario, while it is not always possible to order uncertain outcomes this way.

Dotmaps (Figures 8, 9) are a way to show a discretized representation of uncertainty at each point on the map. We use groups of colored grid cells to represent the distribution of possible air quality estimates at that location (see Figure 6). This idea was inspired by dotplots [13].

Similar static techniques have been proposed, using pixelation on maps to convey uncertainty. Building on previous ideas including using texture and flickering pixels to convey uncertainty, researchers have proposed a pixelated choropleth map to convey uncertainty of a value within counties [28]. Using a monochromatic scale, pixels are assigned colors based on random draws within the margin of error.

We include two different dotmaps in our study. One is an “ordered” dotmap, with cells large enough to discern individually, and ordered within their groups. This might encourage users to compare relative frequencies of colors in different areas of the map. The second version is the “smoothed” dotmap. To create these, we transform each 3x3 group of cells into a 9x9 group of smaller cells with the same percentage of cells per color. These smaller cells are each placed randomly within the outline of the original 3x3 grid. Smaller grid cells may encourage users to visually interpolate the colors to come up with intermediate values. It may be difficult to discern values in highly uncertain areas, since they will look noisy. Previously proposed uses of texture and pixelation for uncertainty in maps sometimes have this goal of obfuscating more uncertain information [28], or using lack of focus to suggest uncertainty [21].

In addition to encoding the ensemble of estimates, dotmaps may also reduce the appearance of boundaries, which have a strong effect on people’s perception of uncertain map data [12]. Reducing firm borders may encourage users to think more about uncertainty [21]. We also want to see if people can use the ordered dotmaps to interpret probabilities, or relative frequencies of outcomes. A similar approach such as stippling may allow visualization design that more finely tunes the tradeoff between clear borders and local details [35].

Risk contour maps (Figure 10) are a hybrid approach, emphasizing the default estimate but highlighting the possibility of the $75^{\textrm{th}}$ percentile. This option presents a less “fuzzy”-seeming view of uncertainty. The discrete boundaries in contour-based maps may have a strong impact on how air quality is perceived; users may be judging the significance of different air quality regions based on the size of each area [36].

To create the maps, we show the median estimate map, and overlay isocontours from the map of the $75^{\textrm{th}}$ percentile estimate. Due to some ambiguity in the contour shapes, we include arrows in these visualizations to indicate the direction of worsening prediction. (For example, in Figure 10, the median estimate for the orange area is shown, while the $75^{\textrm{th}}$ percentile estimate for this area outlined in orange; there is a chance the orange area might be as large as the outlines.) This view shows less uncertainty information than the small multiples or dotmaps, depicting two possible estimates rather than nine. Contours are a familiar representation, so using them to encode areas of heightened risk may be intuitive and help acclimate users to thinking about uncertainty. However, depicting uncertainty by adding discrete boundaries to a map may be misleading by drawing attention to the particular border placement [12].

Applying our study to the real world assumes that people will put effort into understanding their air pollution risk, and would make the same effort in reality. Answers to this study may or may not reflect actions people would take in their lives. To mitigate this, we recruited members of our local community who had been exposed to fire-related air pollution within the past year, corresponding to the sample population in Section 2.4. Personal experience has a significant impact on how people interpret maps, and this group has a relatively small range of personal experiences with air pollution compared with the global population, which may help us hone in on the factors that people use to make decisions with these maps. However, we must be cautious generalizing results to the general public. Of the 17 individuals (age 20-30) in the study group (7 female/10 male), 12 were Computer Science students and 5 were employed in other fields.

The in-person study followed a within-subjects design, where each participant ($n=17$) saw each of five scenarios in each of six map types (30 total stimuli). We use scenario to mean a particular time and location, with all available air quality readings from PurpleAir and government sensors. Each scenario was generated within a bounding box near an urban region in northern California. We chose dates during 2017 and 2018 with significant air pollution from wildfires in these areas (for example, see Figure 11). (Participants were only told that these were real scenarios.) Each stimulus image contained a circle representing the region within which the user would live and be active outside. Each scenario had one circle location that was used for all six map types. The circle locations were chosen to be away from edges and to contain some amount of uncertainty. Each participant used the same computer, which had a large enough screen size for the grid cells in the ordered dotmaps to be discernible.

We first performed a pilot study of two test subjects, using three of the scenarios. In the full study, each user was shown each stimulus, first the interpolation-only view for each scenario, then each other stimulus, in a rotating order so that scenarios were not repeated before seeing each of the others. The order of scenarios and map types was balanced for each user, except for the interpolation-only map type. For example, if a user was assigned the scenario order $S_{0}$, $S_{1}$, $S_{2}$, $S_{3}$, $S_{4}$ and the map type order $M_{0}$, $M_{1}$, $M_{2}$, $M_{3}$, $M_{4}$, they would first see: $S_{0}$ $\times$ interp.-only, $S_{1}$ $\times$ interp.-only, …, $S_{4}$ $\times$ interp.-only. Then, they would rotate through each combination: $S_{0}$ $\times$ $M_{0}$, $S_{1}$ $\times$ $M_{1}$, $S_{2}$ $\times$ $M_{2}$, $S_{3}$ $\times$ $M_{3}$, $S_{4}$ $\times$ $M_{4}$; then $S_{0}$ $\times$ $M_{1}$, $S_{1}$ $\times$ $M_{2}$, and so on.

Before the study, each user was guided through the same presentation which showed and explained each of the map types for a sample scenario; they were allowed to ask clarifying questions before starting the study.

For each stimulus, users were asked three questions:

• •

  Q1. If you had plans to run/bike outside today, would you reduce your plans? User answers on a 7-point scale from “strongly disagree” to “strongly agree.”

• •

  Q2. Where, if anywhere, would you go for relief? User clicks on a point on the map.

• •

  Q3. How confident are you in your answer? User answers on a 5-point scale from “not at all confident” to “strongly confident.”

Users were asked to think aloud as much as possible while answering; we transcribed these answers and then identified different types of phrases from among the responses. Q2 was included to elicit more discussion from users about their decision-making process. Next to each stimulus, a legend appeared showing the colors corresponding to each AQI level (good, moderate, unhealthy for sensitive groups, unhealthy, very unhealthy, hazardous). Each of 17 users answered the questions using all six map types for all five scenarios, for a total of 510 observations.

For $P$, the model included scenario ($S$), map type ($M$), and the interaction between scenario and map type ($S\ast M$) as fixed effects; we included individual ($I$) as a random effect to account for the fact that an individual’s responses to different stimuli are not independent. The sample included 510 responses (17 users $\times$ 5 scenarios $\times$ 6 map types).

We found that scenario ($df=4,F=113.5,p<0.0001$), map type ($df=5,F=12.1,p<0.0001$), and scenario $\times$ map type ($df=20,F=3.1,p=0.0001$) all had highly significant effects on $P$.

To differentiate the effects of map types—ones that include uncertainty versus ones that do not—we performed a post-hoc least-squares mean contrast, finding that $P$ was significantly lower for the Interpolation and Interpolation + Sensors views than for the other four views ($F=50.9,p<0.0001$). That is, including uncertainty in a view led to a higher reduction in physical activity. Within the uncertainty map types, the Ordered Dotmap and Small Multiples led to lower $P$ responses than the Smoothed Dotmap and Contours views ($F=7.54,p<0.0063$) (Table I).

Still, the effect of map type was highly dependent on scenario. Specifically, in one scenario, which showed extremely high air pollution levels, individuals chose to reduce their activity regardless of map type, leading to a significant effect of $S\ast M$. Prior work suggests that the risk level has a higher effect on users’ responses than the visual features of the map, such as contours, focus, and how risk is encoded [21].

To check that this analysis is robust to deviations from normalcy, we used a matched pairs non-parametric Wilcoxon Signed Rank test. For a given individual in a given scenario, responses to Q1 ($P$) for the Interpolation and Interpolation + Sensors map types were, on average, significantly lower than responses for the other four views (Test Statistic: $-883,p<0.0001$). In 54/85 cases, mean participant $P$ response to the non-uncertainty map types was lower than it was for the four uncertainty map types, compared to only 23/85 times where the other pattern was observed (and 8/85 where they were equal). This test showed a less significant difference between $P$ responses for the Ordered Dotmap and Small Multiples views vs. Smoothed Dotmap and Contours views (Test Statistic: $-442,p=0.05$).

These results suggest that our six visualization designs can be categorized into two, or perhaps three, types, each with different effects on users’ decisions (see Figure 12). The first type (group A) includes the standard map types: interpolation only and interpolation with sensors. Users were most willing to continue their exercise outside when judging air quality based on these maps. The second group includes the uncertainty views, which may be broken into two categories. Group $B_{0}$ includes the frequency-framing designs: small multiples and the ordered dotmap. (Though smoothed dotmaps are created with the same frequency information as the maps in group $B_{0}$, their resolution makes it difficult to discern frequency.) Users’ responses to the map types in group $B_{0}$ were intermediate. The maps in group $B_{1}$ (risk contours and smoothed dotmap) are the non-frequency-framing uncertainty views. Users reduced physical activity the most in response to these maps.

The consistency of responses for $P$ also varied depending on map type. We used a loglinear variance modeling approach and found that in a model including scenario and map type as mean effects, and map type as a variance effect, map type had a significant effect on the variability of a participant’s $P$ responses ($df=5,\chi^{2}=29.4,p<0.0001$). Variability of responses was highest for the Interpolation + Sensors view, followed by Interpolation only. (In section 6.3, we discuss reasons this might be the case.)

We created a similar generalized linear mixed model to analyze users’ confidence in their responses ($C$):

While scenario still had a highly significant effect on $C$ ($F=5.3,p=0.0004$), map type showed a less significant effect ($F=2.3,p=0.041$), and scenario $\ast$ map type was not significant ($F=0.9,p=0.61$). Using a Least Squares Mean Contrast, we found that the Interpolation and Interpolation + Sensors views led to significantly lower confidence than the other views (F=9.8, p=0.002), but there was no significant difference between Ordered Dotmaps and Small Multiples vs. Smoothed Dotmaps and Contours ($F=0.005,p=0.95$). Wilcoxon Signed Rank tests confirmed a significant difference between Interpolation and Interpolation + Sensors versus the uncertainty map types (Test Statistic$=-532,p=0.016$).

Users’ confidence was significantly higher for the uncertainty views than for the two standard views (see Figure 13). One caveat is that because we are not considering the “correctness” of users’ answers, we cannot correct for the “hard-easy effect” of confidence reporting [39]. Users often deliberated for longer while making their decisions for the uncertainty views, perhaps resulting in users perceiving these decisions as more difficult and therefore reporting higher confidence ratings for these map types.

We analyzed think-aloud transcriptions to understand how users made decisions with each of the map types. We transcribed users’ think-aloud feedback during the studies, then identified types of phrases or reasoning that came up repeatedly, grouping them into four categories:

Figure 14 shows the number of occurrences of each of these categories per view. Several patterns emerge: