Do You Trust What You See?
Toward A Multidimensional Measure of Trust in Visualization

The Oxford English Dictionary defines trust as “firm belief in the reliability, truth, ability, or strength of someone or something.” Alternatively, personality psychologist Rotter[36] defines trust as a personality trait that describes one’s “generalized expectancy that the oral or written statement of other people can be relied on.”[36] Both highlight the common expectation of reliability as an integral component of trust. In contrast, sociologists view trust as a social construct, classifying it as a group property rather than an individual one [20]. Through this lens, Luhmann [22] argues that “familiarity is the precondition for trust and distrust,” reasoning that one must be familiar with the object of trust. Similarly, Simmel [45] asserts that when faced with something unfamiliar, we “gamble” rather than “trust”. Simmel aligns with behavioral economists’ view of trust as “the mutual confidence that no party to an exchange will exploit another’s vulnerabilities” [40]. This notion of confidence, particularly in decision-making under uncertainty, appears in several of the prior works [4, 8, 12, 7]. Another example from Morgan and Hunt [28] defines trust as the perception of “confidence in the exchange partner’s reliability and integrity.”

Researchers have proposed assessing trust as an enduring trait in people and studying trust as a broad construct that complements personality models such as the Big Five Inventory [39, 9]. For instance, Evans and Revelle created and tested a 21-item scale that measures interpersonal trust, with items such as “Listen to my conscience” and “Avoid contact with others” [11]. Further, research in organizational psychology proposed several dimensions of trust, including integrity, reliability, and credibility, based on a comprehensive review of existing literature on trust measurement [27].

We also considered work from the Artificial Intelligence (AI) and Human-Robot Interaction (HRI) communities to inform how we measure trust in information visualization. Like with visualization, the effectiveness of incorporating AI into organizations is significantly dependent on users’ trust in the technology [17].

In AI, trust can be defined as the willingness of an individual to rely on and be vulnerable to the actions of another party based on the expectation that the other will perform a particular action that is important to the trustor, irrespective of the ability to monitor or control that other party [15, 24]. Researchers measure trust in AI through various means, including user reactions to technological features, perceived usefulness, ease of use, and trust as a predictor of technology acceptance [14, 33]. Although the antecedents of trust in AI are still not fully understood, researchers have identified several dimensions of trust that are relevant in the context of AI, including tangibility, transparency, reliability, task characteristics, and immediacy behaviors [15].

We observed more progress in the HRI community, where researchers have developed various methods for measuring trust. For example, some studies ask participants to rate their agreement to statements such as “I trust that the robot can perform the task successfully” [35, 44]. Others have sought to create standardized and generalizable scales for measuring trust, such as The Reliance Intention Scale [23] and The Trust Perception Scale-HRI (TPS-HRI) [42]. Work by Ullman and Malle and their Multidimensional Measure of Trust (MDMT) [46] is also notable. They began by surveying the cross-disciplinary perspectives of trust, summarizing definitions from dictionaries, human-automation, human-human, and human-robot interaction research. They found that dictionaries typically mention terms such as ability, reliability, honesty, and integrity, and argue that trust in HRI should capture four dimensions: Reliable, Capable, Ethical, and Sincere.

Trust is key in determining whether a user will rely on and build on the information provided in a visualization. Researchers of visualizations acknowledge the complexities of trust and its evaluation, and they employ various metrics to measure trust [10, 41, 47, 48]. This ambiguity in measuring trust makes it difficult to assess the validity of these different measures and whether they capture trust. Without a shared understanding of what trust is and how it can be measured, it becomes challenging to evaluate trust, hindering the ability of researchers to make meaningful conclusions across studies.

The most common approach to measuring trust is directly asking users to indicate their level of trust using a Likert scale [47, 18, 48, 31]. Kim et al. [18] used a 100-point scale to measure participants’ trust in the accuracy of the data. Similarly, Zhou et al. [48] used a 9-point scale to measure the users’ trust towards predictive decision-making. In addition to direct measures of trust, such as asking users to rate their level of trust in a visualization, researchers have also used proxy measures, such as substitution variables. For instance, Xiong et al. [47] presented participants with various map visualizations for a hypothetical firetruck emergency and asked them to choose the most trustworthy one. They also collected ratings of perceived trust and transparency, which encompassed four key dimensions: the accuracy of the information, the clarity of the communication, the amount of information disclosed, and the extent to which the shared information reflected the underlying data. These methods allow for a more nuanced understanding of how users perceive the trustworthiness of visualizations.

Moreover, Mayr et al. [26] provided a definition of trust that emphasized the importance of reliability and understanding for visualizations. They characterize trust as the users’ tendency to rely on and build on the information displayed in a visualization. This concept emphasizes that a user’s trust is intimately related to their impression of the veracity and usefulness of the information supplied. Therefore, visualizations that are perceived as reliable and easy to understand are more likely to be trusted, while those that are perceived as deceptive and complicated are less likely to be trusted. These considerations highlight the importance of developing an effective approach to measuring trust in visualizations.

To examine the relationship between the visualization designs and trust dimensions, we recruited 500 participants from Prolific. They were all English-speaking from the USA and received $5 for completing the study. 255 self-identified as males and 240 as females, with a self-reported age range of 18 to 77. We used 410 visualizations from the MASSVIS dataset [6]. We selected this dataset because the visualizations were collected from a diverse set of sources (e.g., government, news venues, scientific publications), and the visualizations were pre-coded with labels indicating features such as whether they included human-recognizable objects and the data-ink ratio.

Procedure and Design     The experiment was divided into two sections: (1) Asking participants to label the visualizations and (2) Measuring visualization literacy using the Mini-VLAT. The first section asks participants to label 30 visualizations randomly selected from the corpus of 410 visualizations using a 5-point Likert scale based on statements mentioned in Table 1. An attention check was included to ensure engagement. In the second section, participants performed the Mini-VLAT [32] to measure their visualization literacy and explore potential relationships between their responses and different dimensions.

Participants had unlimited time to complete the study, spending an average of 77.3 seconds ($\sigma=98.1s$) viewing each visualization, with a minimum time of 10.1 seconds. Using Pearson’s $r$ coefficients, the correlation test revealed no statistically significant correlation between the time taken to view a visualization and the participants’ Likert scale ratings for each dimension ($p>0.1$ in all cases). Therefore, the duration of participants’ viewing did not impact their ratings or perception of different visualization dimensions.

Data     The variables for this experiment are:

• •

  trust dimension: { familiarity, clarity, credibility, reliability, confidence} $\in[1...5]$

• •

  visualization literacy $\in[0...12]$

• •

  categories: { scientific, government, news, infographics }

• •

  attributes Black & White $\in[yes,no]$; Number of Distinct Colors $\in[1,2-6,\geq 7]$; Data-Ink Ratio $\in[good,medium,bad]$; Visual Density $\in[low,medium,high]$; Human Recognizable Objects $\in[yes,no]$; Human Depiction $\in[yes,no]$.

Results     We began by examining the relationship between the visualization categories and participants’ subjective ratings based on the five trust dimensions. We ran Kruskal-Wallis tests for each dimension to examine differences in subjective ratings according to the visualization source. Except for familiarity, there was a significant relationship between the visualization’s source and the dimensions, all with $p<.001$. Post-hoc Mann-Whitney tests using a Bonferroni adjusted alpha ($\alpha=.0083$) uncovered noteworthy patterns. For example, scientific visualizations yielded significantly lower ratings than all other categories in clarity, credibility, reliability, and confidence. We observed similar patterns with visualizations from government agencies, especially for clarity. Conversely, visualizations from news media elicited significantly higher credibility, reliability, and confidence scores than all other categories. This trend can be seen in Figure 1, and we provide more analysis details in our supplementary materials.

We then examined the relationship between visual features and trust dimensions. Again, we conducted Kruskal-Wallis and post hoc Mann-Whitney tests for each dimension with the appropriate Bonferroni-adjusted alphas. When examining the # of distinct colors and data-ink ratio, we found no significant association with familiarity, credibility, reliability, and confidence. However, we observed a significant difference in clarityrating # of distinct colors and data-ink ratio attributes. Similarly, black & white visualizations received significantly lower ratings for clarity, reliability, and confidence compared to colorful ones. In general, colorful and higher data-ink ratio visualizations were favored.

Finally, we observed a low positive correlation (r = 0.31, p $<$ 0.001) between the participants’ ratings on clarityand their visualization literacy scores. This provides suggestive evidence that higher visualization literacy was correlated with understanding the message conveyed by the visualizations. This finding highlights the importance of considering the audience’s visualization literacy when designing and evaluating visualizations.

Exp. 2 aims to understand how people describe trust in the context of visualization design. Although the dimensions from Exp. 1 were curated from a diverse set of existing literature, we also use this study to examine how well the observed rating aligns with perceived trust. To achieve this, we selected six visualizations based on each of the dimensional values obtained from Exp. 1. Specifically, we chose the visualizations with the three highest and three lowest mean scores for the given dimension, provided that they had at least 17 respondents (top 10 percentile) and a standard deviation that was lower than the median. In other words, after filtering for the visualizations with high response rates and low variance, we chose the visualizations that were, on average, ranked to be the best and worst for the given dimension. This resulted in 30 visualizations (six for each dimension). We then examined how trust values were related to the individual dimension values for these selected visualizations.

Procedure and Design     We recruited 15 participants using convenience sampling, all of whom had bachelor’s degrees, with nine self-identified males and six self-identified females. We used neutral language to avoid unintentional bias; the information sheet and questions excluded any mention of the five dimensions from Exp. 1. We asked participants to “Rank the following visualizations based on how much you trust them, with 1 being Strong Trust and 6 being Strong Distrust.” Then we asked them to “Please explain your ranking.” Each participant ranked five sets of six visualizations corresponding to the trust dimensions from Exp. 1. The orders of the options and questions were completely randomized.

Results     Figure 2 suggests that familiarity and clarity ratings from Exp. 1 strongly aligned with the perceived trust from Exp. 2, with the top three visualizations in both categories eliciting high trust ratings and the bottom three receiving low ratings.

For confidence and credibility, two visualizations from both the agree and disagree categories were trusted and distrusted, respectively. However, in the case of reliability, we observed no obvious separation in the trust ratings.There could be several reasons we observed a different pattern for reliability, e.g., the differences in our study populations. The concept of reliability is complex and multifaceted, and individual differences may affect interpretation. However, our experiment design traded control of individual differences for controlling learning effects and biases.

To analyze the explanations given by the participants, we used an open-ended, qualitative, and iterative approach in identifying the factors used by the participants in ranking the images based on their perceived trust towards the visualizations. Two authors coded the responses and identified seven keywords with mutual consensus: familiarity with the topic, interpretability, reliability, source present or not, visual aesthetic, type of visualization, and accuracy. The authors achieved an inter-rater reliability of 0.947 in the first round and finalized the keywords in the second round. Notably, 12 of the 15 participants said source was the primary reason for trusting visualizations. For example, P12 stated, “Ranking is based on [a] combination of source provided and the nature of visualizations.” Additionally, 10 of the 15 participants explicitly referred to their familiarity with the topic as an important factor in their trust rankings. For example, P15 stated, ‘‘my familiarity with the topic and data source” as the rationale for trust. For the remaining keywords, visualization type was mentioned by 8 out of 15 participants, interpretability was pointed to by 7 out of 15, visual aesthetic by 5, reliability by 4, and accuracy by 3. Interestingly, 5 of the 15 participants considered complex and long infographics untrustworthy visualizations. For example, P10 stated “I find long images with cartoons deceiving” and P13 said “I find long images less trustworthy”. Additionally, participants referred to infographics as “long images”, “images with cartoons”, and “marketing posters” instead of using the technical term, indicating a gap in familiarity with the type of visualization and correctly naming them. These results suggest that researchers should consider these dimensions when measuring trust in data visualizations, as they align with the participants’ explanations.