When Refreshable Tactile Displays Meet Conversational Agents: Investigating Accessible Data Presentation and Analysis
with Touch and Speech

The growing availability of data has allowed decision-making to become increasingly data-driven in virtually all professions. Employees are now expected to use tools like Excel or Tableau for data-driven decision making, and to employ a mix of analytics and visualization. The use of data visualization is, however, problematic for BLV people, and we are witnessing a surge of research interest in investigating accessible alternatives (e.g., [36, 33, 38, 37, 17, 60, 74]).

One approach is to provide a written or spoken description of a graphic. Researchers have investigated what makes a good description [29], how to automatically generate a description from a graphic or answer questions about a graphic (see [18, 26] for recent reviews), or how to generate descriptions directly from data [70, 50, 51]. Some researchers have explored how to incorporate these capabilities into systems designed to support data analysis by BLV people [61, 2, 32].

Another approach is to use non-speech audio, known as sonification, to convey information. For instance, the SAS Graphics Accelerator uses sonification and tables to provide access to visualizations [58]. Sonification can be used as an alternative format to data visualizations, or alongside data visualizations [35]. Recently, there has been interest in combining speech with sonification [24, 64, 65].

The third approach is to provide raised line drawings known as tactile graphics, which are encountered by many BLV students at school. Transcription guidelines recommend their use for accessible provision of maps, diagrams, and graphs, rather than textual or verbal descriptions, in particular if the reader needs the information to participate in discussions or complete a task [44]. Most tactile graphics are produced using swell paper or embossing [56]. However, printed tactile graphics are not well-suited for interactive data exploration, because of the cost and speed of production. Therefore, there has been considerable research into displays that combine auditory and tactile feedback as alternatives to tactile graphics, which include touch screens with audio labels and with/without vibration feedback (e.g., [20, 40]). However, previous research suggests that exploration is faster and more accurate with tactile graphics than with touchscreen-based approaches [8].

A promising alternative is large refreshable tactile displays (RTDs) [66, 69]. Most consist of a grid of pins controlled by electro-mechanical actuators [69]. Their great advantage over traditional tactile graphics is that it takes only a few seconds to display a new graphic. In other words, an RTD is the tactile equivalent of a computer monitor, and like a monitor, it supports interactive data exploration.

Until recently, RTDs have been prohibitively expensive. However, the DAISY Consortium’s Transforming Braille Project has spurred the development of cheaper devices, which are now entering the market. These include the DotPad [13] and Monarch [3], which provide 2,400 (60$\times$40) and 3,840 (96$\times$40) pins. Other devices on the market include the Graphiti [46] (2,400 (60$\times$40) pins) and Metec [39] (6,240 ($104\times 60$) pins), which offer several pin heights and touch-sensitive pins.

Research into the use of RTDs has focused on the display of static images in the fields of art [21] and book diagrams and illustrations [34, 43, 48]. Other work has looked at the display of dynamic images, such as sports matches [45] and animations [22]. RTDs have also been used to show maps for building a cognitive model pre-travel [6, 28, 41, 59, 71], or as mobility aids updated in real-time to show position/obstacles as a BLV person moves through an environment [5, 72, 73].

There has been almost no research into the use of RTDs for interactive data visualization. One paper describes the co-design of tools for the interactive exploration of set diagrams and parallel vectors on an RTD [14], and a poster at VIS 2023 examined stakeholder perspectives of how RTDs could be used to improve access to graphics, including data graphics [23]. There has been more research into systems that use RTDs with audio labels to display maps automatically generated from web-based map systems, which support search, zooming, panning, and route finding (e.g., [59, 73, 71]).

A limitation of RTDs is their low resolution when compared with traditional tactile graphics [47, 22]. This makes the provision of braille labels problematic, and is one of the reasons why audio labels are provided on most RTD map applications (e.g., [59, 73, 71]). Indeed, in their review, Butler et al. [8] found that audio labels on tactile graphics are generally preferred to braille labels and lead to faster performance. Furthermore, it is recommended that tactile graphics be accompanied by a spoken or written description [44].

Advances in question-answering interfaces and conversational agents suggest that we can go one step further and combine an RTD with a conversational agent. We know of no prior research into such a combination. However, there has been some research into combining conversational agents with tactile maps [9], tactile graphics [19], art pieces [27, 10], and with 3D printed models [62, 54, 53]. Shi et al. [62] determined that in addition to audio labels, 3D printed models should allow BLV users to ask questions using conversational language. Reinders et al. [54] found that BLV participants desire multimodal interactions that combine spoken dialogue, touch gestures, and haptic vibratory feedback; this finding was followed by design recommendations that heavily focused on conversational interactions [53].

A Wizard-of-Oz (WOz) study [12] is a research method where an end user interacts with an interface that, to some degree, is being operated by a ‘wizard’ who fulfills an interaction functionality that is yet to be fully implemented [31]. WOz studies often involve eliciting interactions from end users, with the wizard simulating system behaviors that complete their interactions [68, 11, 63]. WOz enables researchers to observe the use of speculative interfaces or artefacts, the outcomes of which can influence future designs.

WOz studies have been used in various domains and contexts. Salber and Coutaz [57] demonstrated how WOz can be extended to the analysis of multimodal interfaces, and formulated a set of requirements for a generic multimodal WOz platform. Kahn et al. [30] employed a WOz technique to control some of a humanoid robot’s speech and actions to investigate patterns in human-robot interactions. In information visualization research, Walny et al. [67] exploited the WOz method to investigate pen and touch interactions for chart creation and manipulation on interactive whiteboards. Recently, researchers applied WOz to investigate BLV users preferred interaction preferences with multimodal systems combining 3D printed models and conversational agents [63, 54]. Kim et al. [32] conducted a WOz study to understand if and how Question Answering systems can help BLV users’ visualization comprehension.

Our WOz study explores a system that combines a conversational agent with an RTD to support BLV people’s interactive data analysis. Our objective is to propose initial design guidelines and to meaningfully engage BLV people in the design of such a system.

We recruited 11 BLV participants (Table 1) from our lab-managed participant contact pool and through mailing lists of BLV support and advocate groups. Participants were aged between 21 and 77 ($\mu$ = 44, $\sigma$ = 17.2). Eight participants identified as totally blind, while three were legally blind. Onset of vision loss was congenital ($n=7$) or occurred during childhood ($n=1$), adolescence ($n=2$), or adulthood ($n=1$). Participants varied in their experience with tactile graphics: five reported substantial exposure to and confidence using tactile graphics, and three reported some use but lacked confidence, while others reported either limited ($n=1$) or no exposure ($n=2$).

All participants regularly used conversational interfaces, including Google Assistant ($n=10$), Siri ($n=9$), and Alexa ($n=5$). These interfaces were commonly accessed on smartphones ($n=11$) and smart speakers ($n=10$), followed by smart watches ($n=7$), computers ($n=7$), tablets ($n=5$), and television ($n=3$). Additionally, several participants ($n=5$) mentioned some use of ChatGPT.

Participants self-reported their technology comfort level using the Technology Adoption Cycle scale [55]. Most ($n=10$) were Early Majority and one was Late Majority. All other categories, Innovator, Early Adopter, and Laggard, were absent.

Participants were also asked to self-report their level of comfort with basic and complex data manipulation/analysis tasks on a five-point Likert scale. Most participants were comfortable with basic tasks (e.g., creating a budget): Strongly Agree ($n=5$), Agree ($n=4$), and Neutral ($n=2$). Comfort with complex tasks (e.g., performing a statistical analysis) was more mixed: Strongly Agree ($n=1$), Agree ($n=4$), Neutral ($n=3$), Disagree ($n=1$), and Strongly Disagree ($n=2$).

For the study we used Graphiti [46] (Figure When Refreshable Tactile Displays Meet Conversational Agents: Investigating Accessible Data Presentation and Analysis with Touch and Speech(b)). The Graphiti’s display has 2,400 (60$\times$40) pins, each of which can be actuated to one of four heights. The Graphiti includes keys to navigate between graphics, and the display can render Braille text. When connected to a computer, a PC application can be used to create or adjust graphics, with changes appearing in real-time on the RTD.

As our work is the first utilizing an RTD for data presentation, we decided to focus on basic data understanding. We selected univariate time series and spatial data, and visualization types we believed our participants would be most familiar with. We selected datasets from finance and climate–Stock, Water  and Fire–that we believed would be both timely and of interest to participants [32], and created several visualizations (line charts, bar charts, and an isarithmic map) (Figure 1).

• •

  Stock gives the price of a stock over four and a half years (Figure 1(a)). It was presented using a line chart, with years on the horizontal axis and price on the vertical axis. This was shown first to familiarize participants with the device and its capabilities.

• •

  Water gives the rainfall and water storage levels in the state of Victoria, Australia between the years 1995 and 2022. The water storage and rainfall information were presented as line charts, either separately (Figure 1(b)-(c)) or together as a multi-line chart with dual axes (Figure 1(d)), depending on the participants’ preference.¹¹1We are aware that dual axes time series may be visually misleading, though for a counter-argument see [15]. However, the limited resolution of the RTD effectively precludes the use of line chartss stacked above one another, which is the most commonly recommended alternative. We therefore decided to provide our participants with the choice.

  Participants could also request to change the presentation format from line chart to bar chart (Figure 1(e)). Line charts are better suited to showing trends, while bar charts are better suited to discrete comparisons. We therefore decided to allow participants to change the visual encoding.

• •

  Fire gives changes in the number of days with dangerous fire weather across Australia between the years 1951 and 2022, using an isarithmic map (Figure 1(f)). Data points are charted using four pin heights on a map of the country: Height 1 (the shallowest) represents a yearly decrease in fire weather, Height 2 represents no change, Height 3 a yearly increase of between 1 and 10 days, and Height 4 (the tallest) an increase of between 11 and 20 days.

All materials were developed with feedback from an experienced tactile transcriber. All visualizations included a braille title and legend. Due to the limited resolution of the Graphiti (in common with other RTDs), braille axis labels and markers were not provided.

User study sessions took place at a location convenient to each participant, either at their residence or at the researchers’ university campus. One or two researchers were present during the experiment.

User study sessions were video and audio recorded. All captured footage was transcribed.

Our aim was to capture and categorize the way in which our BLV participants chose to interact with the system. To achieve this goal, we classified each interaction based on the combination of modalities used (touch, conversation, or touch gestures), as well as their order and intent. One researcher derived an initial set of patterns by classifying the interactions of the first user study session (P1), which were discussed and confirmed with a second researcher. This set was supplemented with additional patterns identified in subsequent user study sessions (P2 - P11). Whenever new patterns were added, all previous sessions were reassessed. On completion, the second researcher validated a subset of the classified interactions, with a subsequent meeting held to reconcile conflicts. The final set of interaction patterns is presented in Section 3.

A thematic analysis was performed similarly on (1) comments made by participants during a session and (2) participant responses to the semi-structured interview. An initial set of codes was devised based on the researchers’ observation of the first user study session (P1). Two members of the research team independently coded this session and met to compare codes, which were then refined and extended. All subsequent user study sessions were then coded by one researcher, with the second researcher validating a subset of the coded data. An additional meeting was held to discuss and reconcile any coding conflicts. Whenever new codes were added, combined or modified, all previous sessions were reassessed. The complete set of coded data was examined and analyzed to generate a set of themes. Three researchers met to review the themes, which were then refined and consolidated. The final set of themes is described in Section 3.4.

Multiple participants controlled the system (referred to from now on as ‘Graphy’) using commands based on prior experience with devices, including smartphones, smart speakers, and computers.

To access the visualization title, six participants preferred to touch read a braille label, e.g., “I would touch the top and feel for braille” (P6). Graphy provided an overview of each visualization upon first touch; to repeat the overview, seven participants asked the device, e.g., “Hey Graphy, can you describe the chart?” (P3).

When extracting detail on demand, participants either performed touch gestures ($n$ = $5$), e.g., single and double tapping to extract descriptions of values at the location at which the gesture was performed, or combined touch gestures with conversation ($n=6$), e.g., tap and hold on a point and ask a question, expecting the device to use their location in conjunction with the question.

For zooming, six participants used the pinch-to-zoom gestures used on their smartphones, e.g., “I am so used to the iOS gestures, can I do that?” (P8), while four combined gestures with voice, e.g., “Hey Graphy, zoom on the area I am holding” (P7). For panning, six participants used swiping gestures from smartphones, e.g., swiping to the left or right with one or more fingers.

For operations that do not cleanly map to smartphone gestures, participants gravitated towards asking the device, or in some cases, using the RTD’s keyboard. Most participants filtered, transformed, added/removed data, and reversed operations through conversation, e.g., “Hey Graphy, only show areas of the isarithmic map where fire risk has decreased” (P3). Several participants chose to use the RTD’s braille keyboard, some of whom indicated that it would be better to be able to connect a QWERTY keyboard, e.g., “I want to type $\ldots$ like when I interact with ChatGPT” (P6).

All participants combined touch, touch gestures, and natural language queries to interpret and analyze the data presented in the visualizations (Figure 2, Table 2).

We identified nine distinct modality interaction patterns that were used during exploration and analysis tasks based on the modalities – Touch, Touch Gesture, or Conversation – and the order in which these modalities were used (Table 3). We distinguished between directed touch, where the participants purposefully explored a graphic, and undirected touch, which was more haphazard, and appeared to lack a purpose. The choice of interaction pattern strongly depended on the type of task and prior tactile experience. The presentation format (line chart, bar chart, or isarithmic map) did not appear to influence the interaction pattern and choice of modality.

Participants asked Graphy’s conversational agent a total of 202 queries ($\mu=18.5,\sigma=15.9$), across four categories:

• •

  Data Extraction (102, 50%) queries related to the retrieval of data values, e.g., “what is the value [that I am touching]?” or “what was the rainfall in 2006?”

• •

  Operations (69, 34%) queries to manipulate the data or visualizations, e.g., “can you zoom this area [of the graph]?” or “can you transform this data into a bar graph?”

• •

  Overview (20, 10%) queries to summarize the data, e.g., “describe this chart” or “what is the trend showing on this chart?”

• •

  Understanding Visualization (11, 5%) queries that provide context relating to the type, layout or encoding of a visualization, e.g., “how many segments are in this graph?” or “what is a bar graph?”

We asked participants to indicate their level of agreement with statements about whether the combination of the RTD and the conversational agent would make them more comfortable completing basic data analysis tasks and complex tasks, and also whether this combination offers significant benefits over each format alone—conversational agent, RTD, or tactile graphic (5-point Likert scale). We ran binomial tests to determine whether the difference between the number of participants who agreed or strongly agreed and those who were neutral or below is statistically significant. We performed Holm-Bonferroni corrections for multiple comparisons [25].

Most participants agreed or strongly agreed with the statement about basic data analysis tasks ($k=8$, $n=11$, $p=0.029$), and more participants agreed or strongly agreed when asked about more complex tasks ($k=10$, $n=11$, $p=0.003$). In addition, participants showed a strong preference for the combination over just a conversational agent ($k=11$, $n=11$, $p<0.001$), RTD ($k=9$, $n=11$, $p=0.012$), or tactile graphic alone ($k=10$, $n=11$, $p=0.003$) (Figure 3).

A thematic analysis of participant responses during the WOz study and interview provided further insights, yielding the following themes: usefulness and use cases, multimodality, deeper experiences through tactile visualization, autonomy, user choice, and design feedback.

P8 described how it would help “reduce the fear and anxiety of [working with] graphs and charts,” suggesting they would like to use Graphy to access the railway heat maps they encounter at work. P1, who works as a data analyst, noted that the device would “make their life a whole lot easier,” and suggested that it could be used to explore stock prices and perform technical analyses. P11 indicated that Graphy could display many of the charts and tables used in payroll duties.

Five participants shared stories of being under-equipped to participate in data literacy activities during schooling, and six suggested that access to a device combining an RTD and a conversational agent could have expanded tertiary opportunities, including statistics (P4, P1, P8), mathematics (P5), computing algorithms (P2), and psychology (P10).

Seven participants spoke of how Graphy could improve access to data for personal uses. This included personal finance and budgeting (P3, P7), monitoring and interpreting health data, such as blood sugar and blood pressure (P7), real-time sports data (P6), and weather forecasts (P9). P7, who became blind at 15, described the utility of Graphy well, “This is great. I would love a real world application of this. There is a lot of data I don’t have access to that I did say 10 years ago.”

P7 discussed how the conversational agent allowed them to “connect the dots between what [they were] touching,” while P6 preferred the combination that Graphy offered compared to printed tactile graphics, indicating that “the conversational agent allows me to get answers to my questions or have the confusion cleared up, while the printed page will remain silent and force me to guess [or move on].”

The Fire isarithmic map, in particular, helped participants build a ‘visual image’ in their head. P8 felt that after the session, the map would “be stuck in my mind, I can still visualize [the state of] Queensland and Australia,” and continued, “my brain is [so] stimulated at the moment.” P8 later discussed how the data would have less meaning to them if it were presented in another format, like a table. P1 became visibly excited, while P7 expressed emotion touching the isarithmic map, describing “I love the shape of Australia $\ldots$ that is something you really miss when you cannot see. Simple things like a map.”

Three participants (P1, P2 and P6), all of whom had higher tactile experience, indicated how the tactile graphics supported independent interpretation. They did not want to simply “be told the answer,” but instead preferred to build their own understanding of the data first through touch. P1 repeatedly acknowledged that they could just ask Graphy for results, but first wanted to explore the visualizations and work out their own answers. P6 described their process, “my first instinct is to use my hands to first feel what the graph is doing… I would like to find the answer myself, then use voice to verify it, you feel more accomplished if you find [it] yourself.” P2 even became visibly frustrated when Graphy gave too much information for their question, feeling that their independence of interpretation had not been respected – “I didn’t ask for an analysis… I just wanted to know what the data was… that would be the type of analysis I would [like to] do myself.”

Three participants discussed how their use would depend on the purpose of the data analysis. P1 described that it would “depend on what I am doing… how deeply I want to explore the data,” stating that in an educational or work context they would engage in deeper interactions to ensure they completely understood what was graphed. P2 discussed varying interactions based on the effort required, “I am more likely to use the conversational agent for more complex tasks, [those where] the effort to benefit ratio is better,” while P8 stated that for basic tasks they would rely on touch, but for work tasks would use touch gestures and voice interaction.

Four participants mentioned that the way they interacted was a personal choice. P4 mentioned that “these are my needs, other people might think and interact differently,” while P5 stated that “the [methods] are all useful, but I prefer buttons and touch.” P3 connected their preference to level of ability, indicating that “because of paralysis, my hand can get sore, so designing it so I can use voice interaction is really important,” while P8 favored voice interaction as “it is what I am most comfortable with.”

Unsurprisingly, all participants described how environment might impact modality choice. P8 and P10 indicated that they would be hesitant to talk to Graphy while working in close proximity with co-workers, and suggested the need for headphones in these scenarios, while P6 said that in loud environments, they would “type commands, but at home [I] would talk to it.”

Prior experience with technology also played a role. This was strongest when it came to willingness to use touch gestures and Graphy’s conversational agent. Participants with extensive experience with smartphones appeared more willing to incorporate touch gestures. P6, who chose to zoom into the visualizations using a double tap and hold, discussed how their choice “corresponded with iPhones.” On the other hand, P8 described preferring interacting using voice, “I think I am so accustomed to my smart speakers and getting information that way.” Two participants who did not use touch gestures often, but were comfortable relying on conversation during interactions, stated, “I am used to Google Assistant” (P4) and “it is easiest to get the information I want… [by] asking a question” (P9).

Four participants (P1, P2, P6, and P8) felt that there was a limit on the complexity of visualizations that can be effectively understood on Graphy’s RTD. This related to the number of pins and pin heights that the RTD currently supports. Prior to being introduced to the Water line chart, participants were asked if they wanted to display water catchment storage, rainfall, or both data together. Only two participants (P1 and P3) chose to show them together when completing the analysis tasks, while three participants (P5, P8, and P10) speculated that displaying both datasets would prove overwhelming. Indeed, when asked to perform the adding data set operation, P8 reported difficulty interpreting sections of the multi-line chart where rainfall and water storage overlapped.

P8 discussed how the RTD’s pin heights were easy to read in isolation, e.g., in a six dot braille cell, but difficult to read in close proximity to one another, and P6 said that in close proximity, the pin heights did not have enough ‘contrast.’ P1 and P2 focused on the low pin resolution of the RTD, describing the isarithmic map as a ‘pixelated version’ of Australia. P2 also felt that Graphy was capable of rendering visualizations at lower resolutions than what they were used to with tactile graphics.

Keyboard input was also requested by three participants, to allow shortcuts, or as an alternative to speech input and touch gestures when manipulating visualizations, e.g., filtering or transforming.

Five participants shared stories of frustration when being under-equipped to participate in data literacy activities during school years, highlighting the need for accessible data tools. This aligns with prior research showing that BLV students are underrepresented in STEM disciplines in tertiary education, due to unequal access [7].

Participants’ overall reaction toward the combination of RTD and conversational agent was very positive. They indicated that the combination would support both basic and more advanced data analysis. We observed that all participants used a mixture of touch, touch gestures, and natural language queries in their interactions with data and visualization. There was also a clear preference for the combination of RTD and conversational agent over a conversational agent, RTD, or printed tactile graphic alone.

To some extent, this preference for the combination of RTD and conversational agent is to be expected, given that multiple modalities increase adaptability [52]. In line with this, our participants indicated that different modalities would allow them to cater to personal preferences and the task context and purpose.

However, our results also revealed that prior experience with tactile graphics played a key role in the choice of modality. In particular, participants with lower tactile graphic experience leveraged the conversational agent to a greater degree. This aligns with earlier research by Reinders et al. [54], where participants with lower tactile experience used a conversational interface embedded in a multimodal interactive 3D printed model earlier and more frequently than participants with higher experience. Nonetheless, all participants, even those with little tactile graphic experience, strongly preferred the combination of RTD and conversational agent over a conversational agent alone. Furthermore, one participant (P8) stated that over time, they may be more willing to use touch more frequently and confidently, suggesting that less experienced BLV users may build tactile confidence and adjust their choice of interaction after using the system.

We were surprised by the strength of preference for tactile visualizations by users with more tactile experience. Participants spoke about how a tactile graphic allowed them to visualize the data, and most of those with greater tactile experience did not want to simply “be told the answer,” preferring to assemble their own understanding of the data through touch first. This desire for independent understanding came in spite of the additional time required for tactile exploration. Independence of interpretation has been recognized as a critical factor for increasing confidence in data-based decision-making by BLV people [16]. For experienced tactile graphics users, the RTD provided BLV users with the opportunity for individual sense-making, just as visual graphics do for sighted users, in a manner that is complementary to information provided by a conversational agent.

A key question raised by any WOz study is whether the interface desired by participants can be constructed using current technologies. Because we catered for a broad range of queries pertaining to summary and background information about the visualizations (informed by [32]), almost all participant queries could be fulfilled using the scripts we prepared. The generation of these scripts was straightforward, and we believe could be readily supported by existing conversational agent platforms such as Dialogflow²²2https://cloud.google.com/dialogflow and Rasa³³3https://rasa.com/docs/rasa/.

One exception was the Fire isarithmic map, when participants were asked to determine the number of fire-weather days for a specific city. One participant (P10) asked for the location of the city and was given directions generated by the wizard on the fly. Another instance occurred when P10 was exploring the map and asked the agent if there had been a change in dangerous fire weather in a small regional city they had once visited while holidaying. Answering questions such as this requires the conversational agent to have an understanding of background information about the data. In principle, this may be supported by means of large language models (LLMs), but it raises issues of reliability and trust.

Participants also desired control over the flow of speech output, including the ability to pause, resume, or stop the conversational output. Placing users in control during conversational interactions has been identified as a critical need with mainstream conversational agents [1, 4]. Based on our experience designing conversational agents with BLV co-designers [53], this would be straightforward to support using Speech Synthesis Markup Language⁴⁴4https://www.w3.org/TR/speech-synthesis11/.

Our study also suggested that participants valued the use of pin height and single-touch and multi-touch gestures. Unfortunately, cheaper RTDs, such as the Dot Pad, do not support variable-height pins, and are not touch-sensitive. Even touch-sensitive RTDs, such as the Graphiti or Monarch, only support single-finger touch gestures and require the user to lift their other fingers when completing gestures. Thus, how best to support touch gestures remains an open issue.

One area that needs further research before we can build an actual system, is a better understanding of how to design interactive tactile visualizations for an RTD. While there exists well-established guidance on visualization best practices when it comes to different types of data and tasks [42], such guidelines do not exist for the design of tactile visualizations for RTDs.

For instance, at present, we do not know the relative effectiveness of tactile line graphs and bar charts for different tasks, or the affordances they offer in this context. In our study, we observed that participants tended to prefer bar charts over line graphs. P6 described “the bar chart is easier to follow, I think it [allows me to] more accurately feel data at any particular point.” This seemed to be particularly true for participants with lower tactile experience, with P9 stating that “[with] the bar chart [it] was a lot easier to determine changes, [both the] highs and lows.” We observed that participants tended to trace line graphs using one or two fingers, with many tracing back and forth between inflection points, and that they traced up and down individual bars in a bar chart, and that in some cases, two hands were used to trace multiple bars. This hints that, just as for visual line graphs and bar charts [42], tactile line charts are suitable for exploration of trends, while bar charts are suitable for discrete comparisons. However, more research is required to test this.

While there are guidelines for the design of traditional tactile graphics (e.g., [44]), there are significant differences between these and tactile graphics displayed on RTDs. One important difference is the current low resolution of RTDs, and the consequent inability to use braille labels, e.g., markers and values on axes. Participants identified possible solutions, but more research is required.

Another area requiring research is how best to support zooming and panning. Touch gestures were used by six participants to pan (swipe) and zoom (pinch). However, when panning or zooming, five participants (P1, P2, P6, P7, and P8) became confused and lost their location when the RTD was refreshed. P1 said “Wow… that is crazy… I lost where I was.” P7 spoke of how “when zooming, [you need] to not lose your frame of reference, to be able to find it again.” P2 directly suggested that a scroll bar could be used to avoid this, “if you are panning a graph you could have an indicator where spatially in the graph you are, for example, you could have one column and row of pins, if you’re viewing the bottom then the column would be lower. Like a scroll bar would show where you are in a page.” Previous research with RTDs has suggested the use of buttons, rather than gestures, for operations like zooming to avoid context loss and user confusion [73, 71]. More research is needed into ways in which multimodal feedback can be used to retain user understanding of position and context during interactions such as panning and zooming, filtering, and search.

A further interaction warranting investigation is using pin actuation to highlight points or regions of interest on the RTD. Actuation has previously been used to highlight routes on maps [22]. We did not support actuation in our study but actuation could, for instance, support search and multimodal output where data points referenced in audio output are concurrently actuated in the visualization. This would be analogous to the use of speech with haptic vibratory feedback in interactive 3D models [53]. Such multimodal output would have been particularly useful with the Fire isarithmic map, when a participant asked for the location of a city and was given directions from the agent, which would have been more effective if combined with actuation.

As the first step to understand the benefits of the combination of an RTD and conversational agent, our study involved only three visualization types – line graphs, bar charts, and isarithmic maps – and four task types – exploration, value identification, extremum identification, and trend. Future research needs to consider a broader range of visualization types and examine additional tasks users perform with these visualizations. We also primarily focused on the presentation of univariate spatial and time series data. Visualizations of more complex multivariate data, e.g., multiple time series visualizations, should be investigated.

As the majority of our participants were totally blind, we are also interested in recruiting additional legally blind participants, specifically those who still have residual vision, as their perspective may differ from that of people who are totally blind.

Our findings should also be interpreted within the limitations of the WOz method. While our findings strongly suggest that a combination of RTD and conversational agent is an effective tool for data analysis, it is crucial to implement and evaluate an actual system with BLV users. It would also be important to test the system with participants over a longer period of time and in real-world settings, e.g., in-the-wild studies in schools and workplaces.