DataTV: Streaming Data Videos for Storytelling

The research agenda for visual analytics called for a focus on the production, presentation, and dissemination of analytics results [28]:

• •

  The case for communication: Viégas and Wattenberg remark upon the proclivity of visualization for communication by virtue of its graphical form, and encourage focusing on so-called communication-minded visualization [30] where communication enables collaborative analysis.

• •

  Embedded dissemination: To reach its full potential, communication capabilities should be integrated into the visualization tools themselves [28]; for example, Tableau now incorporates the story points feature [18], and most commercial tools supports exporting interactive dashboards to the web.

• •

  Literacy and casual viewers: Presenting insights from data to the masses requires taking the visualization literacy [7] of everyday users into account. Thus, the notion of “casual visualization” [24] is important.

Storytelling is the conveyance of a sequence of events involving characters and places—stories—and has a history spanning thousands of years [25]. A visual narrative is a story told primarily using visual media, such as illustrations, photographs, animations, video, and—now—visualization [12, 27].

• •

  Richness in visuals: Visual forms of communication, including icons, illustrations, schematics, photographs, and full-motion video, have long been considered to be one of the most compelling storytelling formats [8, 27].

• •

  Video tutorials: Complex skills are today often learned through step-by-step instructions on the internet [15]. Several projects have endeavored to increase the richness of such video tutorials: ToolClips [15] provide contextual access to video assistance, Pause-and-Play [23] links tutorials to the real application, MixT [9] automatically generates video from demonstrations, and a recent approach makes software tutorial videos interactive [21].

• •

  New storytelling formats: Beyond traditional providers (HBO, MSNBC, ABC, etc) as well as the new breed (Amazon Prime, Hulu, and Netflix), open video sharing sites such as YouTube have democratized access to visual storytelling. With such openness comes entirely new storytelling formats that were previously infeasible, such as vlogs (video blogs) [14], reaction videos [3], and video commentaries, including the ever-popular Let’s Play videos [31].

• •

  Live streaming: An additional recent development in internet video is the increased focus on live, so-called streaming, content, where the media is received and presented to the consumer at the same time it is delivered by the provider. This development was particularly driven by live gameplay content, where prominent gamers—called streamers—broadcast their game screen on services such as Twitch.tv. While streaming is also used for heavily produced events such as eSports tournaments, where professional gamers compete over thousands of dollars in prize money in games such as Overwatch, League of Legends, and Dota 2, the vast majority of live streams on Twitch and elsewhere are created using specialized broadcasting software such as OBS, XSplit, GameShow, etc. Such software are what inspired Data TV, and are designed for creating high-quality streams using an optimized user interface that can be controlled during live broadcast by a single user.

Combining communication-driven visualization with storytelling yields data-driven storytelling: narrative techniques for telling stories about data [26].

• •

  Storytelling in visualization: Gershon and Page first proposed using storytelling for visualization [13], and their work has since been followed up by workshops [10, 11], surveys [16, 26], and even commercial tools [18].

• •

  Multiple storytelling media: The purview of data-driven storytelling has quickly grown, from dashboards and slideshow presentations [18] to more esoteric formats such as sketching [20], journeys in time and space, and even comics [4, 17]. In their survey of narrative visualization, Segel and Heer [26] identify seven genres—magazine style, annotated charts, partitioned posters, flow charts, comic strips, slideshows, and videos—and also suggest that future storytelling approaches may combine genres.

• •

  Data videos: Amini et al. [1] recently identified data videos as motion graphics combining both sound and visuals to tell a data story. Pointing to prominent examples from the New York Times and the Guardian, their work encourages professional storytellers to use visuals to craft their narratives. Their follow-up Data Clips [2] work is an authoring tool for creating data-driven clips incorporating visualizations that can be assembled into longer data videos. While obviously deeply influential to our work, their treatment focuses on the careful and deliberate production of data videos through ideation, sketching, storyboarding, capture, and editing; live streaming is not covered or even considered. Furthermore, DataTV provides support for including additional content from the streamer’s computer, such as video, sound, music, and existing applications (such as Tableau or Spotfire), merely using the ability to capture the user’s full desktop.

The DataTV tool is modeled along the standard workflow used in game streaming software such as OBS, XSplit, and GameShow, where the streamed output of the tool is managed using the concept of sources and scenes:

• •

  Source: Streaming input such as from a window, specific application, audio source, or multimedia content.

• •

  Scene: Composition of sources on an empty display that can be recorded and streamed as output from the tool.

DataTV operates in one of two distinct modes: (a) offline mode, where the user configures sources and scenes, or (b) live mode, where scenes are recorded (and possibly streamed). Most operations (creating, modifying, and deleting scenes and sources) can be performed in both modes to allow for responding to live events (e.g., adding a new web-based visualization to the stream on a spur-of-the-moment idea), but the most common workflow is as follows:

1. 1.

   Preparation: The user prepares all scenes and sources for recording in offline mode. This involves creating the sources, creating the scenes, and composing the sources for each scene on the output display canvas.

2. 2.

   Recording: The user switches to live mode, sets up the stream settings (if enabled), and begins the recording. If the video is streamed live, the user cannot easily switch back to preparation; instead, any changes to scenes or sources must be made while recording. If the video is merely being recorded and not streamed, the user can stop recording and go back to preparation, allowing the clips to be edited together at a later stage.

3. 3.

   Storytelling: The user creates the data video in live mode. This entails switching between scenes (by selecting the scene to display in the scene manager), managing specific sources (transforming, toggling on and off, annotating, etc), interacting with visualizations and other windows, and potentially narrating and/or capturing webcam video of the user.

DataTV maintains a list of currently available sources in the source window (Figure 2). Using this widget, sources can be created, toggled on and off, and deleted in both offline and live modes. Sources are created by selecting the source type and then associating the source with the appropriate object, such as a specific browser window on the desktop. Supported sources include the following:

• •

  Window capture: Video output from a specific window on the desktop based on the window title, class, or executable.

• •

  Video capture: Video output from an external device, such as a webcam or video camera.

• •

  Audio capture: Audio output from a microphone.

• •

  Media objects: Sources based on static images, video files, and rich text.

A scene in DataTV is an empty canvas containing sources that form the current output. Only one scene can be active at a time in DataTV; the active scene is displayed in the main composition window (Figure 2) and governs what is recorded and streamed when in live mode. The user can switch scenes using the scene list.

Managing a scene essentially entails managing the sources involved in the scene. In offline mode, most scene management operations are “heavyweight” in that they require significant setup that is not amenable to live recording. Examples of such operations include adding sources to the scene, managing their depth order (governing the drawing order of the sources), and transforming them (translating, scaling, rotating, and cropping). These operations are achieved by interacting with the sources in the composition window.

In live mode, users should mostly use “lightweight” operations that are designed for easy interaction while recording. This includes toggling the visibility of sources, zooming and panning in a source using mouse dragging and the mouse wheel (such as to zoom in on a particular part of a visualization or window), and scribbling using the annotation feature.

Sportscasters regularly use annotation to scribble symbols and highlights directly on the video feed, such as to explain specific events in an instant reply of a touchdown or goal. Presentation software such as Microsoft PowerPoint supports similar “ink annotations” where the presenter can draw pen and highlight strokes directly on a slide to illustrate a specific point. DataTV supports a similar video annotation feature through its live mode interface (Figure 3), which provides a pen, highlighter, and eraser tool. The interface also allows the user to select the drawing color as well as to clear all of the annotation from the output when moving to a new scene.

Our DataTV prototype supports recording to standard native video file formats (FLV, MP4, MOV, etc) as well as live streaming output to services such as YouTube and Twitch.tv. The tool also provides full control over stream settings for both video and audio output.

We implemented our DataTV prototype based on OBS (Open Broadcaster Software) Studio, an Open Source live streaming package for multiple platforms. Our extensions were made in C++ and significantly modifies the workflow of the tool to include a streamlined live mode interface, including scene management, zooming and panning, and live video annotation.

The first example uses the Keshif [32] system¹¹1http://keshif.me/ to explore the “Nobel Prize Winners” dataset. Keshif is a visualization system designed for interacting with multi-dimensional data using a sophisticated faceted browser. The key interaction in the Keshif system is linked selection, which is a generalization of brushing and linking supporting highlighting, filtering, and comparison.

The Nobel Prize Winners dataset is composed of the basic profiles of all winners, including their pictures, year of winning the prize, nationality, etc. The overall story of this example is to briefly introduce South Africa, particularly its former leader and Nobel Peace Prize winner, the late Nelson Mandela. The analyst uses a YouTube video of Mandela (Figure 4) giving a speech as the introduction. Live video used in this way will make the presentation engaging and draw in the viewer. It is easily achieved in DataTV using a YouTube source and controlled live by the user; no specific off-line editing is needed. Then we turn off the video and use a live window with Keshif to lead the story from introducing the categories of Nobel Prize to their age distribution. Again, the storyteller can do this in real-time simply by interacting with the Keshif visualization in a normal web browser window, potentially narrating his findings using the webcam and microphone. The DataTV platform will capture all of these sources, compose them in real-time, and stream them to a remote server. Compared to static media types, such as infographics, the ability to use an interactive interface enables the analyst to change the topic and approach during the storytelling. For example, potentially in response to an audience question through the Twitch chat service, the analyst may decide to give a little history of the Nobel Peace Prize, as well as Nelson Mandela’s term as president, culminating in him winning the Nobel Prize. We then conclude the session with a short YouTube video of him giving another speech.

For this example, we use TimeFork [5], an interactive visual prediction technique to support users exploring the future of time-series data. The TimeFork implementation allows an analyst to explore a multitude of potential futures for specific stocks by initiating a dialogue between the analyst and the user. Here we use TimeFork to create a narrative for tech market stocks (here Apple and Netflix).

Our user starts off the session with an introductory scene involving a YouTube video featuring Warren Buffett talking about the current stock market. Then the user switches to a scene incorporating the TimeFork tool, using it to predict the current trend of hot tech stocks Apple and Netflix by simply interacting with the tool in a web browser. The user can even switch to a window of a desktop visualization tool such as Tableau or Spotfire and include that into the video if desired. Meanwhile, the user is narrating the interaction and the findings using a live recording of himself using the computer’s webcam and its built-in microphone. All of these media sources are composed, recorded, and streamed in the DataTV tool in real-time using the active scene specification. The main narrative of the data video would describe a scenario for stock market trends, similar to what you may hear on financial news. The user closes the video with a PowerPoint information slide that summarizes the main trends.

The CommentIQ [22] system is designed to help community moderators manage large amount of comments associated with online articles by automatically ranking them based on criteria such as relevance, readability, personal experience, and length.

In this example, the user wants to author a streaming data video about the community response on an article from The New York Times²²2http://www.nytimes.com/ titled “City Reacts: State of Emergency” during the 2015 racial unrest in Ferguson, Missouri following the death of Michael Brown at the hands of a white policeman. The user collects two infographics with topics on murder rate across races, and the SWAT deployment rate of different races. The online interactive visualization CommentIQ looks deeper into the comments of the article from The New York Times.

This data video leverages the advantages of each of the three types of media source: video, infographics, and visualization. First, the narrator uses a YouTube video of the Ferguson incident as the introduction. This video shows the confrontation between the protesters and the police, providing a suitable framing to the video that emphasizes the direness of the situation. The two infographics (Figure 9) give background information by showing an overview of the guns and crimes in the area. Finally, the CommentIQ visualization allows the narrator to discover trends in how the NYT commenter community responded to the article. In all of these cases, the narrator can scribble directly on the composited video output to highlight interesting aspects of the video, such as outliers or trends.

The live streaming functionality of the DataTV platform opens up an entirely new potential for the New York Times to provide a live complement to go with their online comment system. Using the DataTV live stream, community moderators could aggregate and discuss comments in real time, for example when polling voter panels for political debates.

We engaged two volunteer participants—Expert 1 and 2—to evaluate our system. The participants were Ph.D. students in the field of HCI or information visualization with at least three years of experience. Both were male.

We conducted the experiment on a standard laptop computer equipped with a 15-inch LCD screen (resolution 1280$\times$800), a standard keyboard, and a three-button mouse. The built-in camera was used for recording the user’s speech with voice.

Each participant’s task was to create data videos using DataTV and the data visualization randomly assigned to them. Participants were required to use at least one interactive visualization in their video. They were allowed to pick any other appropriate media sources on the web to support their stories. The tasks were devised so that the experts would need to explore the visualizations, select media sources, and sift through information to create a narrative. The data stories we asked them to create were inspired by Section 5:

• •

  Obama Budget: A visualization illustrating the components of government budget in the year 2013.

• •

  U.S. Census: Demographics for the state of Florida.

A set of tasks was prepared for each scenario, requiring the participants to explore the data visualizations in detail and to look for supporting information online, before finally creating a data video to answer the question. A sample set of tasks for one of the scenarios (Obama budget) is given below:

• •

  Explain roughly how the total ($3.7 trillion) were allocated (using a 30-second video);

• •

  Explain the main types of spending (using a 30-second video);

• •

  Explain how the spending has changed since the last budget (using a 40-second video); and

• •

  Describe how much of the budget was allocated to social security (using a 60-second video).

Participants were shown a demonstration of DataTV and given as much time as they needed to explore and familiarize themselves with the system. We then gave them a sheet of tasks and asked them to create a data video in response to each task. The resulting data video would ideally make use of multiple media types, interactive visualization, and innovative storytelling techniques. During the creation process, which was capped at 60 minutes, the experts were allowed to ask questions about the interface. We followed a think-aloud protocol with the participants, and recorded their behavior via video and written observations.

For the purpose of this study, the participant-generated stories were recorded and saved (instead of streamed dynamically). The final participant-generated videos were limited to a duration of one minute. During the process, the participants used the DataTV tool to design, sketch, record, and produce their videos. After completing their tasks, we followed up with an interview where participants explained their process and provided feedback on the system.

We collected and analyzed both the products as well as the observations and interview feedback from each expert review session.