Quda: Natural Language Queries for Visual Data Analytics

Our research is motivated by recent progress in V-NLIs, which provides engaging and effective user experience by combining direct manipulation and natural language as input.

Some research emphasizes on the multi-modal interface that interacts with users using natural language. Back in 2010, Articulate (Sun2010) identifies nine task features and leverages keyword-based classification methods to understand how user’s imprecise query weighted in each feature. Sisl (Cox2001) is a multi-modal interface that accepts various input modes, including point-and-click interface, NL input in textual form, and NL using speech. Analyza (Dhamdhere2017) is a system that combines V-NLI with a structured interface to enable effective data exploration. To address the ambiguity and complexity of natural language, Datatone (Gao2015) presents a mixed-initiative approach to manage ambiguity in the user query. Flowsense (Yu2020) is a V-NLI designed for the dataflow visualization system. It applies semantic parsing to support natural language queries and allows users to manipulate multiple views in a dataflow visualization system. Several commercial tools, such as Microsoft Power BI (microsoft), IBM Watson Analytics (IBM), Wolfram Alpha (wolfram), Tableau (tableau), and ThoughtSpot (thoughtspot), integrate V-NLIs to provide better analytic experience for novice users.

Another line of research targets the conversational nature of V-NLI. Fast et al. proposed Iris (Fast2018), a conversational user interface that helps users with data science tasks. Kumar et al.  (Kumar2016) aimed to develop a data analytics system to automatically generate visualizations using a full-fledged conversational interface. Eviza (Setlur2016) and Evizeon (Hoque2018) are visualization systems that enable natural language interactions to create and manipulate visualizations in a cycle of visual analytics. Similarly, Orko (Srinivasan2018) is a prototype visualization system that combines both natural language interface and direct manipulation to assist visual exploration and analysis of graph data.

The aforementioned approaches are mainly implemented using rule-based language parsers, which provide limited support for free-form NL input. Instead of generating visualizations, Text-to-Viz (Cui2020) aims to generate infographics from natural language statements with proportion-related statistics. The authors sampled $800$ valid proportion-related statements and built a machine learning model to parse utterances. Although promising, Text-to-Viz does not support queries for a broad range of analytic activities.

The performance and usage scenario of V-NLIs highly depend on language parsers. The cutting-edge NLP models have reached close-to-human accuracies in various tasks, such as semantic parsing, text classification, paraphrase generation, etc. However, few have been applied to visualization-oriented V-NLIs. We argue that the release of the high-quality textual dataset would assist the training and evaluation of NLP models in the domain of data analytics.

An increasing number of studies in the visualization community have employed supervised learning approaches in various scenarios such as mark type classification (Savva2011), reverse engineering (Poco2017), color extraction, etc. The capability of these approaches, however, highly relies on the availability of massive datasets that are used for training and evaluation (Lee2019).

Some datasets consisting of visual diagrams are constructed and open-sourced for visualization research. For example, Savva et al.  (Savva2011) compiled a dataset of over 2,500 chart images labeled by chart type. Similarly, the Massvis dataset (Borkin2013) contains over 5,000 static chart images, and over 2,000 are labeled with chart type information. Beagle (Battle2018) embeds a Web Crawler extracting more than 41,000 SVG-based visualizations from the web, and all are labeled by visualization type. Viziometrics (Lee2018) collects about 4.8 million images from scientific literature and classifies them into five categories, including equation, diagram, photo, plot, and table. Poco and Heer (Poco2017) compiled a chart corpus where each chart image is coupled with bounding boxes and transcribed text from the image. Instead of compiling a corpus containing chart images, Quda includes queries accompanied by analytic tasks for visualization-oriented V-NLIs.

Viznet, on the other hand, collects over 31 million real-world datasets that provides a common baseline for comparing visual designs. Though containing rich textual information, Viznet can hardly be applied to provide training samples for learning-based NLP models. On the contrary, Quda is designed for helping V-NLIs understand utterances describing analytic tasks in information visualization.

Our corpus is primarily designed for helping V-NLIs classify queries into multiple analytic tasks. Therefore we survey related corpora for multi-label text classification, which is the task of categorizing a piece of textual content into $n$ possible categories, where $n$ can be $0$ to $n$ inclusive. The website (multilabel) lists a large number of datasets for multi-label text classification. We selectively introduce four datasets that are comparable to Quda. For example, the 20 NG or 20 Newsgroups dataset (Lang1995) includes about $20,000$ documents collected and categorized from $20$ different newsgroups. Ohusumed (Joachims1998) contains $13,930$ medical abstracts belonging to $23$ cardiovascular disease categories. The IMDB dataset contains $120,919$ text summaries of the movie plot, and each is labeled with one or more genres. The TMC2007 dataset contains $28,569$ aviation safety reports that are associated with $22$ problem types. Different from the aforementioned datasets, Quda is the first corpus targeting queries in V-NLIs.

Table 1 shows the statistical comparison between Quda and the aforementioned datasets. Inspired by the prior work (Moyano2017), we use five metrics to characterize each dataset. Besides the number of instances and categories, we also compute cardinality (i.e., the average number of categories for each instance), density (i.e., the cardinality divided by the number of categories), and diversity (i.e., the percentage of category sets present in the dataset divided by the number of possible category sets.) Detailed definitions and calculations are described in (Moyano2017). We notice that Quda is comparable with other corpora in most metrics. Especially, Quda has the highest density among all corpora, which means instances in Quda have higher probability of being assigned with multiple categories.

To understand how data analysts raise questions from data tables, we recruit 20 individuals, 10 females, and 10 males, with ages ranging from 24—31 years ($\mu=25.45,\sigma=2.80$). In our study, we identify “data analysts” as people who are experienced in data mining or visual analytics and have at least one publication in related fields. Most participants ($16/20$) are postgraduate students majoring in Computer Science or Statistics, and the rest are working as data specialists in IT companies. All of them are fluent in English and use it as working language. The experiments are conducted on a laptop (2.8GHz 4-Core Intel Core i7, 16 GB memory) on which participants read documents and create queries.

Tasks play a vital role in authoring queries. Our study begins with the taxonomy of $10$ low-level visualization tasks (Amar2005), including Retrieve Value, Compute Derived Value, Find Anomalies, Correlate, etc. The pilot study shows that participants may be confused about some tasks. For example, one commented, “Determine Range is to find the maximum and minimum values in one data field, which is similar to Find Extremum.” To help participants clarify the scope of each task, we compiled a document presenting each task from three aspects, i.e., general description, definition, and example sentences. The document is shown in the supplementary material.

Data tables provide a rich context for participants to create queries. Hence, to diversify the semantics and syntax of queries, we have prepared $36$ real-world data tables covering $11$ different domains, including health care, sports, entertainment, etc. Tables with insufficient data fields may not support some types of queries. For example, assume a table about basketball players has two columns, i.e., players’ name and their nationality. Participants may find it hard to author queries in the “Find Extremum” or “Correlate” categories which usually require numeric fields. Hence, instead of skipping the tasks, we allow participants to revise tables by adding new columns or editing existing ones if necessary. Moreover, we selectively choose data tables that have rich background information and explanation for each column so that participants can get familiar with them in a short time. All data were collected from Kaggle ²²2https://www.kaggle.com/datasets, Data World ³³3https://data.world/, and Google Datasetsearch ⁴⁴4https://datasetsearch.research.google.com/.

The interview began with a brief introduction to the purpose of the study and the rights of each participant. We then collected the demographic information of each participant, such as age, sex, experience in data mining and visual analytics. After that, participants were asked to familiarize themselves with $10$ analytic tasks by reading the document.

In the training stage, participants were asked to author queries for $10$ tasks based on a sample data table, which differs from those used in the main experiment. We instructed them to think aloud and resolved any difficulties they encountered. We encourage participants to author queries with diverse syntax structure and semantics.

The pilot results indicate that the participants were more enthusiastic in generating sentences with diverse syntax if they are interested in the context of the table. At the beginning of the main experiment, we presented $10$ tables with a detailed explanation of the context and data fields to participants and encouraged them to choose two based on their interest. We randomized the presentation order of the $10$ tasks. Participants were guided to author at least two queries given a table and a task, and no time limit was given to complete each query. To summarize, the target queries collected at this stage is $20(participants)\times 2(tables)\times 10(tasks)\times 2(queries)=800(queries)$ However, because some of the participants enjoyed the authoring experience and generated more sentences for some tasks, the total number of queries exceeded $800$.

After the main experiment, participants were asked to re-assign task labels for each query. Because some queries belong to multiple tasks in nature. For example, The sentence “For applications developed in the US, which is the most popular one?” falls in both “Filter” and “Find Extremum” categories. The identification of all task labels is important to reflect the characteristics of each query.

Finally, a post-interview discussion was conducted to collect their feedbacks. Interviews were audio recorded and we took notes of the participants’ comments for further analysis. Each interview lasted approximately two hours, and we sent a $\$15$ gift card to interviewees for their participation.

The result of our interview study is a corpus containing $920$ queries generated by data analysts. The length of queries ranged from $4$ to $35$, with an average of $11.9$ words ($\sigma=3.9$).

We observed that participants found it enjoyable to write queries for these tasks. For example, one participant commented in the post-interview discussion, “Find Anomalies is an interesting task because it requires an in-depth understanding of the data table.” She further added, “I asked ‘Based on the relationship among rating, installs, and the number of reviews, which app’s rating does not follow the trend?’ because this kind of abnormal apps is worth investigating.”

In this step, we aim to acquire at least $20$ paraphrases for each expert sentence. We followed the crowdsourcing pipeline (Burrows2013) to collect paraphrases using crowd intelligence. We developed a web-based system that the crowd can access through MTurk. Our system records both crafted queries and user behavior, e.g., keystrokes and duration, for further analysis.

We describe our task as (Potthast2010), “Rewrite the original text found below so that the rewritten version has the same meaning, but a completely different wording and phrasing” to help crowd workers understand our task. The interface also encourages the crowd to craft paraphrases that differ from the original one in terms of sentence structure. We demonstrate both valid and invalid examples to explain our requirements. The interface displays an experts’ sentence and instructs workers to rewrite it. After a sentence is submitted, workers are allowed to author paraphrases for other expert sentences, skip sentences, or quit the interface by clicking “Finish”. Finally, we use an open-ended question to collect workers’ comments and suggestions for this job. We sent 0.1$ as a bonus for each sentence after validation.

Based on prior work in corpus collection (Lasecki2013; Burrows2013) and the pilot experiments, we established a set of rules in the main experiment to ensure the quality of the collected sentences. First, to collect paraphrases that are diverse in structure, we limited the maximum number of sentences one could create ($20$ in our study) to involve more crowd intelligence. Second, to obtain sentences with high-quality, workers were informed that the results would be reviewed and validated before they receive a bonus. Third, the interface recorded and analyzed their keystrokes and duration in crafting sentences to reject invalid inputs. For example, we rejected sentences crafted within $5$ seconds or with less than $10$ keystrokes. Fourth, to avoid duplicate sentences with other workers, we compared the input with all sentences in the database.

The result of the first step is a corpus containing $18,400$ crowd-generated paraphrases accompanied by $920$ expert-generated sentences. The goal of this step is to filter out paraphrases that are semantically inequivalent compared to the expert sentence.

Due to the ambiguity in human language, asking workers directly to identify whether paraphrases are semantically equivalent to the expert sentence would be restrictive and unreliable. For example, given an expert’s sentence, “How many faculties are there at Harvard University?”, and a worker’s, “How many departments are there at Harvard?”, it is not clear whether the two sentences have the same meaning because “faculty” could mean both “department” and “professor”. Instead of asking whether a paraphrase is semantically equivalent, we asked comparison questions to capture the equivalence strength of one paraphrase with respect to others. To be specific, given an expert sentence, a worker is shown a pair of paraphrases written by the crowd and asked to choose which one has closer meaning to the expert’s sentence. Pairwise comparison is effective in capturing semantic relationships (Parikh2011). However, exhaustive comparisons, i.e., each paraphrase compares to all other paraphrases using a large number of workers, is unnecessary and cost-prohibitive. Hence, borrowing from the idea of relative attributes (Parikh2011), we design a strategy to collect pairwise comparison dataset for all paraphrases, and train scoring functions to estimate the semantic equivalence scores of them.

Assigning all task labels to a query is critical to V-NLIs. Therefore, we conduct an experiment to evaluate how a well-performed multi-label classification model performs on our corpus, and reveal the characteristics of our corpus with different experiment settings.

The multi-label classification model trained with Quda can benefit existing NLI toolkits in task inference for free-form natural language. We choose NL4DV (Narechania2020) as an example, which is the first python-based library supporting prototyping V-NLIs. NL4DV takes NL queries and a dataset as input, and returns an ordered list of Vega-Lite specifications to respond to these queries. Its query interpretation pipeline includes four modules, such as query parsing, attribute inference, task inference, and visualization generation. The task inference module of NL4DV focuses on the detection of 1) task type, 2) the parameters associated with a task, and 3) ambiguities of operators and values. The toolkit currently supports five base tasks (i.e., Correlation, Distribution, Derived Value, Trend, Find Extremum), and Filter. To infer tasks, the authors set rules according to task keywords, POS tags, dependency tree, and data values. Although effective in some scenarios, the rule-based approach is hard to extend and limited for free-form natural language.

We aim to improve the precision and range of task type inference with the help of Quda. We first use Bert (Devlin2019) trained on Quda_expert (details are introduced in Section 7.1) to detect task type. Our model supports ten low-level tasks and can assign multiple task labels for each query. Though our model cannot explicitly detect the Trend task supported by NL4DV, Trend is usually identified as Correlation and Distribution, and appropriate visualizations are recommended. For example, “Illustrate the distribution of the population of Ashley across the five years.” is identified as Distribution and Filter. Second, we setup a set of rules to generate visualization for all tasks. Similar to NL4DV, the ten tasks are divided into $9$ base tasks and Filter. For base tasks, we take into account the type of data attributes and tasks to identify proper visualizations and visual encodings. For Filter and its associated attributes, we highlight information in the Bar chart rather than filter to few bars according to the design guideline (Hearst2019).

Some V-NLIs, such as Eviza (Setlur2016), Articulate (Sun2010), and FlowSense (Yu2020), have embedded template-based auto-completion to improve user experience (Gao2015). That is, after a user query is partially completed, the interface provides hints for possible queries. Though useful, these techniques are tailored to specific applications and grammars, and can hardly be generalized to other V-NLIs. Query recommendation techniques in search engine (BaezaYates2005; Zhang2006; Anagnostopoulos2010) suggest a list of related queries based on previously issued queries, which are independent of applications and have wide usage scenarios. With the help of Quda, we aim to develop a query recommendation technique for V-NLIs, which recommends semantically meaningful queries based on user input, data table, and Quda.