Understanding User Experience of COVID-19 Maps through Remote Elicitation Interviews

Data visualization is becoming an important part of daily life as more and more people commonly encounter data-driven online platforms and tools in contexts ranging from journalism to urban spaces [1, 20, 27, 6]. The SARS-CoV-2 pandemic further increased this importance by putting data visualization in the center of everyday life [3, 23, 30]. In this urgent situation, many organizations started to create visualizations to better explore and communicate the global effects of the pandemic. This created many challenges including designing visualizations with complex data for wide audiences, including experts, policymakers, and lay people with different levels of data literacy.

To understand the impact of these visualizations, it becomes critical to investigate the experiences people have using them while the pandemic is still ongoing. People use these visualizations to learn more about the subject, and inform their decisions about their personal life. Additionally, it is a topic that has a strong emotional aspect. These conditions put COVID-19 visualizations in a unique position where there is an opportunity to learn interesting and relevant information about how people interact with these visualizations in the course of their everyday life, and learn from their experience to drive visualization design decisions accordingly.

For a phenomenon that affects many people deeply on a personal level, we believe it is important to investigate user experiences beyond performance measures to capture the richness of their experience. Traditionally, evaluation methods for visualization are mostly based on usability and performance  [16, 12]. Recent studies have been introducing methods from other fields to capture the richness of the user experience further [4, 7, 15, 9]. Hogan et al. [9] introduced Micro-Phenomenological Interviews for visualization studies and described it as the Elicitation Interview (EI), that aims to capture personal experiences with great detail and with less risk of bias and post-rationalization [26]. They suggest that visualizations can trigger associations with past experiences, therefore affect the user’s experience heavily. We believe COVID-19 visualizations are especially good cases for EI as the reading and usage might be strongly influenced by the user’s personal experience and knowledge.

To understand the user experience with data visualizations during the pandemic, we need to adapt co-located evaluation methods for remote settings, due to the existing mobility disruptions, and because a large portion of the population is social-distancing or even quarantined at home. In the last decade, researchers started to conduct remote interviews to understand their advantages and disadvantages over physical interviews. Remote interviews have many advantages in terms of reducing cost and increasing accessibility by removing geographical and mobility constraints [13]. On the other hand, challenges were reported regarding connectivity and rapport  [29]. Poor internet connection can cause calls or recordings to be disrupted  [5]. While studies found remote interviews supportive for building rapport, they caution against a deceptive closeness between the participant and researcher  [5], and point out that the lack of intimacy in remote interviews may cause problems for sensitive questions  [29]. More recent literature reports that remote interviews resemble in-person interviews in rapport [14]. Remote interviews in the literature span all types of interviews from structured, to semi-structured, to open-ended interviews. We are not aware of any studies that applied the Elicitation interview technique remotely.

In this paper, we describe how we conducted a remote user study on two COVID-19 visualizations with seven participants. We selected the WHO Coronavirus Disease (COVID-19) Dashboard [23] and the COVID-19 Dashboard by the Center for Systems Science and Engineering (CSSE) at Johns Hopkins University (JHU) [30], as both are from reputable sources, present similar types of information, have comparable views, yet use distinct visual styles. In the literature of elicitation interviews used for data visualizations, the method has been applied in a co-located setting with static visualizations[9]. Nowak et al. [21] used the EI method for narrative visualizations and conducted two of the six interviews remotely, yet did not discuss the challenges or advantages. We aim to build upon these previous studies on Elicitation Interviews by expanding the method to a remote setting and applying it to interactive visualizations. With our study, we contribute considerations on how to conduct remote elicitation interviews for interactive visualizations and design recommendations for public health visualizations.

As visualizations become more important for our daily lives, evaluating and understanding how people interact with these visualizations becomes more critical to create better visualization tools. Previous studies have extensively analyzed different evaluation methods for different types of visualization scenarios  [16, 12]. The visualization literature has a growing pool of evaluation methods for different visualization stages and purposes  [19, 16, 12]. Yet, visualization evaluation studies still mostly focus on user performance, through assessing objectively measured metrics like time on task and accuracy  [9, 16, 12].

Some more recent studies move beyond performance-related objectives, including metrics like enjoyment  [4], engagement  [7, 15], memorability  [28, 2], and emotion [8, 32]. Wang et al. [32] argue that researchers typically concentrate on performance, comprehension, or insights when determining the importance of visualizations. In their work, they expand these definitions by emphasizing emotional responses and including different forms of engagement. They propose an alternative way to value data representations: by their creativity and their ability to engage. They expand engagement further by defining different dimensions of it, which are affective engagement, physical engagement through touch and movement, intellectual engagement, and social engagement. In another study focused on visualization engagement, Kennedy et al. [15] explored subjective indicators of engagement through a diary study and a set of focus groups. Some of the indicators were provoking questions, creating empathy, generating curiosity, provoking surprise, changing minds, or provoking a strong emotional response.

There are different methods and metrics to get a deeper understanding of the user’s experience of interacting with a visualization. One of the most common methods is the interview method, which is generally structured, semi-structured or open-ended interviews with a focus on the usability aspects  [16]. These types of interviews fall short when we want to get a deep understanding of the user’s experience at the moment of using the visualization, as they carry the risk of post-rationalization  [9]. To overcome this, recent studies used the Elicitation Interview technique on visualizations  [9, 21]. It is a method of descriptive phenomenology, developed by the psychologist Vermersh  [31], as he was aiming to transfer the implicit knowledge of experts to non-experts. Later on, this method was used with any type of lived experience by the cognitive scientist Varela  [24]. The method was also used in the HCI field for understanding text entry experiences  [17], tactile experiences  [22] and recently in the visualization field to understand the experience of making sense of data displays  [9, 10, 21]. For static and narrative visualizations, the method was found useful as it allows understanding users’ subjective experiences beyond their attitudes and judgments  [9, 21]. In these studies, Hogan et al.’s  [9] focus was on static visualizations while Nowak et al.’s  [21] focus was on narrative visualizations with limited interactivity. However, there have been no studies on more complex interactive visualizations. We believe applying EI to complex interactive visualizations is important, as interactivity is an important element for visualizations.

The importance of remote methods increased as the COVID-19 Pandemic limited the mobility of societies worldwide. Before this situation, research on remote interviews gained speed with the rapid developments in technology and the wide usage of available tools. Online interviews have an immense strength that overcomes temporal, financial, geographical, and mobility constraints [13]. As a result, remote interviews contribute to inclusion and diversity of participants [11]. However, they introduce new types of challenges. One important challenge is connectivity problems [5]. Even though connectivity problems decreased significantly over the years of remote interview literature, even small problems can still affect the flow state negatively for EI. Another challenge identified by several studies [29, 5], is related to rapport. Seitz [29] notes that it is hard to build rapport in remote interviews as there is no physical, in-person relationship. On the other hand, Brown [5] states that remote interviews can create a deceptive closeness, as the tools used (e.g. Skype) are usually used with family and friends. We believe this perception might be shifted due to the widely applied remote working situations. Additionally, a more recent study by Jenner et al. [14] reports that remote interviews resemble in-person interviews in rapport and the depth of information shared.

Hogan et al. [9] uses the Elicitation Interview in a co-located setting and even described how the physical adjustments should be to assist the desired mental state. Nowak et al. [21] conducted two of the six interviews in a remote setting, however, they note that it was done for convenience and did not discuss it further. We believe it is important to understand and highlight the advantages and challenges of remote elicitation interview methods on interactive visualizations.

An elicitation interview requires a mental state that enables the evocation of the lived experience. The participant should feel as if they were living that moment. This state enables acquiring details that would otherwise go unnoticed. To create such a mental state, an elicitation interview uses a special questioning technique and interview structure. The interview is conducted in a quiet place without any distractions. During the interview, open-ended and content-empty questions are asked in the present tense, to evoke the experience. The questions are “What” and “How” questions, no “Why” questions are asked to avoid judgments and post-rationalization. The interviewer observes specific cues to understand if the participant is in an evocation state. Speaking in the present tense, gazing into space, taking pauses, and slurring speech are some of the cues that indicate the participant is in the evocation state [24].

We primarily use Hogan et al.’s [9] detailed elicitation interview guide where the phases and the question types are explicitly framed for visualization studies. The questioning technique has an iterative structure, the level of detail increases through the interview. The questions themselves are content-empty, they don’t introduce new information, but they aim to reveal the actions, sensations, and feelings of the participant at an increasing level of granularity. An important pre-interview consideration is selecting a particular experience. The elicitation interview requires focusing on a singular experience, it is not suitable for general experiences. The common case for visualization studies has been creating that experience by providing users with a visualization to view, either a week before the interview [9], or right before the interview [10, 21].

The interview structure has six phases. The first phase is notifying the participant about the aim of the study, and the nature of the interview, that it can go deeper into experiences than regular interviews. The participant should be aware that they can stop the interview at any point. The second phase is commencing the interview using the phrase, “If you agree, I would like to go back to the time when you started to experience X.” Then the questions of when and where the experience happened are asked. After this, the third phase begins where the evocation state is induced through visual, auditory, kinesthetic, and emotional questions. Some examples of these questions are: “When you are there, what do you see? At this moment, what are you hearing? At that time, what is the position of your body?, At that time, what are you feeling?”. The fourth phase is called the diachronic stage. It aims to understand the temporal structure of the experience, how it unfolded over time. Example probing questions that mark the key moments of the experience are: “How do you start?, What do you do then?, What happens at the end?, How do you know you have finished?” [25].

After the temporal structure is defined, the fifth phase begins. It is called the synchronic structure and it aims to deepen the episodes defined in the previous phase. Thus, probing questions are more fine-tuned versions of the evocation stage (third stage) questions, that aim to reveal visual, auditory, kinesthetic, and emotional sensations in greater detail. At the sixth and last phase, the interviewer reformulates the entire interview and asks the participant if they have anything to add.

Our work expands the previous works on elicitation interviews by applying the method in a remote context. In addition to the elicitation interview about the experience, we asked post-interview questions to understand the implications of the method being remote. We discuss how the non-verbal cues that indicate an evoked state is translated to a remote context. Furthermore, the visualizations we selected are tools where the users can freely explore the collection of visual information. This creates a challenge of understanding the navigation behavior precisely since there are no visual records but only the remarks of the user. We believe that elicitation interviews can still be useful without a visual record, and we discuss the level of insights that this type of information can induce. We see our work as a probe investigating the elicitation interview methodology in a remote setting, and intend this paper to function as a case study. Through describing our methodologies and reflecting on what worked and what did not, we hope to provide some insightful lessons for others to plan and conduct their elicitation interviews remotely, and to benefit future studies.

In this section, first, we will present the themes that emerged from the remote EI. Then we will present our observations from the interviews and results of the post-interview questions about participants’ remote social experiences and their relationship with data during the COVID-19 pandemic. We will use the abbreviations “WHO” for World Health Organization’s Coronavirus Map, and “JHU” for Johns Hopkins University’s website while presenting our results. We will use identifiers in square brackets for distinguishing participants’ statements.

In this paper, we presented a case study on remotely gathering subjective feedback of people’s experience with interactive data visualizations in the context of the coronavirus pandemic. Elicitation Interviews aim to reveal hedonistic, sensory, and emotional aspects of using data visualizations, with previous studies mostly focusing on static or narrative visualizations [9, 21]. We conducted remote elicitation interviews with two major data-driven dashboards. The two systems in our study were complex interactive visualizations about the COVID-19 pandemic data, a global and current topic that affects many people on a personal level. Therefore we believe it is important to understand the user experience beyond performance measures and explore the personal driving factors of open-ended exploration.

Our findings indicate that spatial exploration can be driven by a user’s existing knowledge about global events, curiosity about the user’s local situation, and personal connections with places. For both visualizations, users were commonly interested in the overall picture and refrained from deeper analysis. Users also generally preferred seeing trends over absolute numbers of cases and deaths. Supporting previous findings, we identified color as an important element that enables users to interpret data. Furthermore, we observed that color can induce positive or negative feelings towards the visualized topic.

In terms of conducting Elicitation Interviews remotely, our observations indicate that users felt safe and comfortable with remote interviews due to their increased experience in remote activities for work, school, and social life during the pandemic. However, we also noticed that sustaining an evocation state can be challenging in remote settings. Our work is a step towards understanding the human experience with data more thoroughly, in settings where conducting in-person studies are not possible. We aim to investigate combining remote elicitation interviews with additional objective methods like activity logging or screen capturing in the future.