”They Aren’t Built For Me”: A Replication Study of Visual Graphical Perception with Tactile Representations of Data for Visually Impaired Users

Visual perception research in graphical interfaces began fundamentally with Cleveland and McGill’s study (Cleveland and McGill, 1984). Their foundational work laid the groundwork for understanding how people interpret graphical information. They introduced a systematic way to assess the effectiveness of different graphical representations, which has been extensively replicated and expanded upon in subsequent research. Researchers have adapted the original experiment to explore how specific audiences, such as individuals with disabilities, the elderly, and those with varying levels of visual literacy, interact with graphical data. Notable replications include Heer and Bostock’s work, which applied these principles to crowdsourced environments, showcasing how perceptions can vary widely in less controlled settings (Heer and Bostock, 2010). Additionally, Talbot et al.’s focused exploration of bar chart perception provides further granularity on graphical interpretation (Talbot et al., 2014).

Recent research in visual perception experiments has increasingly focused on creating more accessible data visualizations through the integration of tactile, auditory, and multimodal feedback, addressing the needs of diverse audiences, including the visually impaired and elderly. Studies have explored the efficacy of audio narratives and haptic feedback in enhancing the accessibility of data visualizations (Siu et al., 2022; Fan et al., 2022; Wang et al., 2 06), emphasizing the necessity for designs that accommodate non-visual modalities. Significant work has been done on developing haptic interfaces that enable blind users to interact with statistical graphics on web platforms (Kim and Lim, 2011a; Guinness et al., 2019). Additionally, methodological advancements have been proposed to ensure that socio-technical considerations are integrated into visualization design (Lundgard et al., 2019; Marriott et al., 2021), fostering a more inclusive approach. These efforts underscore a pivotal shift towards visualizations that are not only technically proficient but also universally accessible, highlighting the critical role of inclusive design principles in the advancement of visual perception research (Cherukuru et al., 2022; Sharif et al., 2021; Mishra et al., 2022).

Research in accessible computing interfaces has significantly evolved, particularly focusing on data sonification and tactile feedback to enhance data interaction for users with disabilities. Studies have explored auditory feedback mechanisms that help visually impaired users understand complex data sets (Wall and Brewster, 2006; Zhao et al., 2008; Wang et al., 2 06; Thompson et al., 2023), effectively substituting visual cues with sound. A significant focus has been on the development of sonification systems, such as the importance-driven sonification techniques of the Line Harp (Bru et al., 2023), which enhances line chart accessibility. Similarly, the Erie declarative grammar (Kim et al., 2024) facilitates data sonification, allowing users to perceive complex data through auditory means. Screen reader technologies have also seen innovative applications, notably through plugins like VoxLens (Sharif et al., 2022), which enables interactive data visualization accessibility on the web. Advancements in screen reader technologies have facilitated accessible navigation through data visualizations, converting graphical elements into comprehensible auditory formats (Demir et al., 2010). Furthermore, the integration of multimodal feedback systems combines tactile, auditory, and sometimes olfactory cues to provide a richer interaction experience (Kim and Lim, 2011b). These efforts are not merely supplementary but are crucial for users who rely on alternative sensory channels to access information, underscoring the need for technologies that recontextualize rather than replicate data presentation across various sensory modalities (Choi et al., 9 06). This research emphasizes the importance of inclusivity in technological development and aligns with broader goals of universal design, highlighting how accessible interfaces can expand digital engagement for all users (Marriott et al., 2021). Furthermore, the Data Navigator (Elavsky et al., 2023) serves as a toolkit focusing on accessibility-centered data navigation, proving essential for users requiring non-visual data interaction. These tools not only ensure compliance with accessibility standards but also promote an inclusive approach to data engagement, highlighting the shift from traditional visual data representation to multimodal data interaction that accommodates a wider range of sensory preferences and capabilities.

Tactile graphics enable BOVI individuals to interact with graphical data through embossed or raised diagrams, offering a non-visual approach to data representation. Experiments conducted by Goncu et al. (Goncu et al., 2010) highlighted preferences for tactile diagrams with gridlines and Braille values over direct transcriptions, and tactile tables over tactile charts. Engel and Weber (Engel and Weber, 2017a) further emphasized the importance of tick marks, grid lines, texture, and the positioning of legends for the readability of tactile charts, documenting challenges like information overload and orientation issues in their survey with 71 participants (Engel and Weber, 2017b). Additionally, they explored differences in affordances among various chart types including bar, line, scatter, and pie charts (Engel and Weber, 2018). Watanabe and Inaba (Watanabe and Inaba, 2018) investigated the suitable texture granularity for tactile bar charts on capsule paper through experiments where participants were asked to count the number of bars under different texture conditions. Watanabe and Mizukami (Watanabe and Mizukami, 2018) conducted an experiment comparing tactile scatter plots to tactile and electronic tables, where participants were asked to identify the relationship between two variables. The results showed that tactile graphs outperformed the other two conditions. In a related study with 8 blind and low-vision participants, Yang et al. (Yang et al., 2020) compared tactile node-link and matrix diagrams for tasks such as pathfinding and cluster identification, noting a preference for node-link diagrams, which performed best except in adjacency tasks.

Our first hypothesis, H1, was that BOVI readers of tactile graphics would have the same ranking of the four chart types found in sighted participants, but that the accuracy would be lower. Our data provided only partial evidence for H1. Both midmeans and error confidence demonstrated similar rankings of charts were similar, but in aggregate the accuracies of BOVI readers was similar or even higher in some cases, as with the bar chart. One-sided t-tests could not confirm the hypothesis that the error of BOVI readers was higher than previous studies ($p>0.5$ in comparisons with Cleveland and McGill (Cleveland and McGill, 1984) and Heer and Bostock (Heer and Bostock, 2010)).

Our second hypothesis, H2, was that BOVI readers of tactile graphics would take a longer time of inference than sighted readers of visualizations due to the physical and mental processing required. We compare our results from those reported in Heer and Bostock. Our participants, on average, viewed 59 charts making the total number of readings $n=651$. The average completion time per chart judgment was 26.74 seconds for BOVI users interacting with tactile graphics. In contrast, Heer and Bostock’s MTurk study reported a higher average time of 54 seconds per trial, with a median response time of 42 seconds and a standard deviation of 41 seconds. A one-sided t test finds this data does not support H2. This comparison suggests that our tactile graphics, while designed for BOVI users, enabled relatively faster responses compared to the MTurk study. We note that Heer & Bostock suggest that in a laboratory setting, compared with MTurk, they expect that participants would be faster.

Our results indicated some nuance in the data that was not captured by the first two hypotheses. In particular, the midmeans chart and the aggregate error disagreed on whether the performance of our subjects was better, equivalent, or worse, than results from previous replications of this experiment with sighted participants. A recent work by Davis et al. offers a different type of analysis of graphical perception studies that enables statistical statements about the variance between subjects using hierarchical modeling (Davis et al., 2022). We repeat that analysis with our results to better understand the outcome of our experiment.

To fit our data to a hierarchical model, we need to declare the assumptions of the model. Our assumptions will model how the data was generated, parameterizing the various effects that we believe led to the values offered by our subjects. We assume that these effects are stochastic, and so we model them as random variables, resulting in a Bayesian model. We then use numerical methods to determine the likely posterior distributions for these variables when conditioned on the data generated by our experiment. For a broader overview of this technique, see Gelman et al. (Gelman et al., 2020)

Davis et al. build a hierarchical model by starting with a simple linear model, which assumes that the average performance of each participant is normally distributed about the true mean for the population - essentially that any differences between participants are the result of random chance. They then expand their model by changing different assumptions. These assumptions include restrictions on the output of the model (replacing the normal distribution with the Zero-Inflated beta distribution), an additional random effects term that assumes some individual differences effect between participant and visualization ($U_{\mu}[\text{vis}[i],\text{participant}[i]]$), and additionally learning submodels for the precision ($\phi_{i}$) and probability of zeros($\pi_{i}$).

By adding these additional terms and adding submodels, in sum, the hierarchical model is able to capture a participant having different types of error or bias for each visualization type. We replicate the same model used in that work, including using the same weakly-informed priors as they reported, and then compare our analysis to their study. We note that the number of participants (eleven vs. more than a hundred) and the number of trials per participant (approximately sixty vs. more than a thousand) results in experimental datasets of different scales, and so some of the analysis of the resulting hierarchical models still had high levels of uncertainty. But we highlight two charts that can tell us more about the difference between the BOVI subjects of our study and the sighted participants in the previous study.

First, in Fig 5, we compare the Cumulative Distribution Functions of the hierarchical models of the absolute response error of the participants’ answers. We are able to produce this chart with the hierarchical model from Davis et al. because it calculates a posterior distribution rather than point estimates, letting us look at the shape of the types of error we found in our study. It is notable that the both studies have both the same ranking of chart types consistently through different levels of error, both low (less than 5 absolute error) and high (more than 20 absolute error). At the same time, our study appears to have less errors with very small error (less than 5 absolute error), indicated by the flat region at the very left of the plot. This is consistent with repeated think aloud statements by our subjects that it was very hard to decide on small differences of estimated answers, i.e. 70% vs. 75%.

Next, in Fig 6, we compare the mean absolute error across our eleven participants on each chart type with errors from the Davis et al. experiment. In the parallel coordinates chart, the mean performance of each participant across all 10-20 trials from each of the four chart types is calculated. This chart was used in Davis et al. to show that some participants have different rankings or relative performances across visualization types - one pink line, for example, showed a bar chart being less effective than a stacked bar chart. In our data, on the left, we do see similar variance in the rankings of the four chart types within our eleven participants. It is also notable that the performance appeared to vary broadly in our sample of eleven participants. However, it isn’t clear that the variance noticed in our study was markedly different than the variance evident in Davis et al.

Our hierarchical analysis revealed both similarities and differences with previously reported results on a wider study run on sighted participants within a crowdsourced study. By fitting the data to a Bayesian model we are able to analyze the CDFs and note the the performance of BOVI participants is similarly distributed across true error types, although there is some difference in the distribution of guesses with small error. In addition, the comparison of mean performances per participant per chart type showed that while our participants had noticeable variance in performance and preference of charts, it wasn’t clear if it was different than previously reported results. Both findings suggest that a larger study would be needed to analyze the types of between-subject and within-subject variance that appears in the data, and to compare it with statistical significance to previous studies.

Through think-aloud statements during the experiment, ad-hoc interviews after the experiment with all eleven participants, and the followup group interview with four participants, we gathered qualitative data about the difficulties encountered by our subjects in reading our tactile graphics, as well as the strategies they employed. We conduct a thematic analysis of this data, clustering statements into themes and highlighting quotes from our subjects.

Our findings suggest that while tactile graphics adapted from visual design principles can be beneficial, they may not fully meet the accessibility needs of those born without sight. This aligns with recent studies emphasizing the importance of designing with and for BOVI users to create truly effective and accessible tactile graphics (Zong et al., 2022; Lundgard and Satyanarayan, 2021). As (Reinders et al., 2024) suggests, incorporating conversational agents and advanced tactile displays could further enhance accessibility, pointing towards a multimodal approach as a promising future direction. Our results support this direction, indicating that tactile graphics should be part of a broader, inclusive design strategy that prioritizes user-centric development.

While it produced useful insights, our study had some limitations. The primary constraint is the reliance on existing visual encodings which might not optimally translate into tactile formats. This limitation is evident in our mixed results where some tactile graphics performed well while others did not, indicating a possible mismatch in the tactile adaptation of visual data (Kim et al., 2023). Additionally, our sample size and the diversity of our participant group, though adequate for preliminary insights, may not fully represent the wider BOVI population, possibly limiting the generalizability of our findings.

The design of tactile graphics itself, produced using swell paper technology, introduces another limitation due to the restricted resolution and detail this method can provide. This potentially affects the users’ ability to discern fine details in complex graphs, potentially skewing the accuracy and speed of data interpretation when compared to visual graphs as noted in foundational studies (Cleveland and McGill, 1984). The accessibility and cost of producing high-quality tactile graphics also pose significant limitations. While swell form technology is cost-effective, it is not universally accessible, and the financial burden may be prohibitive for some institutions or individuals, limiting widespread adoption.

Moreover, the diversity of techniques used by participants to interpret the data underscores a limitation in the standardization of tactile graphics. For example, one participant employed a method of using fingers as calipers to measure the length of the bars, while another estimated spatial divisions through mental segmentations. Such personalized techniques, although creative, highlight the lack of a one-size-fits-all approach in the current tactile graphic designs.

Participants also noted difficulties with the texture and construction of tactile elements. Many found the uniformity in the texture of bars confusing and suggested that varying the texture could aid in differentiation. Double encoding, which combines tactile information with other sensory cues like varied textures, was recommended to improve usability and accuracy.

Looking forward, we propose a series of large-scale studies to further validate and refine the tactile graphic designs. These studies should explore a wider array of tactile and multimodal presentation techniques, involving more diverse participant groups to enhance the reliability and applicability of the findings. Furthermore, collaboration with BOVI participants should be emphasized to tailor designs more closely to user needs, potentially involving technologies that integrate tactile feedback with auditory and possibly olfactory cues for a richer user experience (Zong et al., 2022). We anticipate that such inclusive research efforts will be crucial in addressing the broader accessibility challenges faced by the aging population with vision loss (Alliance, 2022).