HybriDialogue: An Information-Seeking Dialogue Dataset
Grounded on Tabular and Textual Data

Given a multihop question from OTT-QA, crowdsourced workers (Turkers) from Amazon Mechanical Turk Crowston (2012) were asked to decompose it into a series of simpler intermediate questions and answers to formulate a simulated conversation in English. ²²2https://confident-jennings-6a2f67.netlify.app/plaid_interfaces/examples/1a_example_1.html As opposed to datasets such as Wizard of Wikipedia Dinan et al. (2018) that are more open-ended, our annotators have a specific goal in mind: to answer an original complex question. By utilizing a single annotator to represent both sides, we keep the flow of the dialogue consistent and natural as it converges to the final answer. The usage of two annotators for our specific task comes with the risk of having one user diverge and reduce the chance of reaching the correct final answer.

We refer to the multihop question from OTT-QA as the “ultimate question”. Turkers are instructed as follows: “In this task, you will engage in a dialogue with yourself. You will act as two characters: the seeker and the expert. At the top of the page, you are given the Ultimate Question. The seeker wants to know the answer to the ultimate question. However, directly asking this ultimate question is too complex. Thus, the seeker needs to decompose (break down) this complex question into a sequence of simple questions, which the expert will answer using a database.” To further emphasize the naturalness of the dataset, Turkers were encouraged to ask questions that required understanding the conversation history context, such as through co-referencing. For example, Turkers used proper nouns with pronouns and indirect references such that they logically refer to their antecedents. An example conversation is demonstrated in Figure 1 and an overview of the dataset collection process is shown in Figure 2.

A conversation is composed of a sequence of turns. Each conversation consists of a minimum of 4 turns and a maximum of 6 turns. This limitation is specified to ensure that Turkers are thoroughly decomposing each complex question and the conversations do not go off on tangents. Each turn $T$ acts as a piece of the decomposition of the ultimate question. The i-th turn $T_{i}$ consists of a natural language question $Q_{i}$, a natural language answer $A_{i}$, a reference $R_{i}$ from an English Wikipedia page, and an available reference pool set ${RP}_{i}$. The Turker provides $Q_{i}$, $A_{i}$, and selects a particular $R_{i}$ from the set $RP_{i}$. $R_{i}$ can be considered the evidence required to generate $A_{i}$ given the question $Q_{i}$. The reference pool $RP_{i}$ contains different types of references including the (linked) paragraph, a (whole) table, a single inner table row, multiple inner table rows, or a single cell.

We differentiate between multiple rows and the whole table in order to obtain a more specific source for the information. For example, the question "Do you have a list of Steve’s accomplishments?" requires a Table response as the answer contains a summary of the table. On the other hand, the question "Did he ever compete in the Grand Prix event type?" requires a selection of specific rows of some table. In order to enforce the naturalness and moderate the difficulty of questions, we restricted ${RP}_{i}$ based on $RP_{i-1}$ and $R_{i-1}$. In other words, the type of questions that the Turker could ask were restricted to the references enabled from previous selections. In the Turker interface, $RP_{0}$ is restricted to the intro paragraph and any whole table references in a provided starting page. We illustrate how reference candidates are added to the reference pool in Figure 3.

To ensure high-quality samples, we conducted various filtering steps. Rejections were made due to the Turker not following the instructions at all or having poor-quality conversations. For example, if the Turker purposefully copy and pasted unrelated paragraphs of texts, repeated the same questions multiple times, used unrelated references, or utilized a single reference throughout the entire conversation, we automatically rejected it. Turkers were paid an average of $1.1 per conversation. Completing a conversation took the worker an average of 5 minutes, which translates to an average of $13.2 per hour. In some cases, we gave bonuses to Turkers who consistently submitted high-quality results. After final verification of the accepted HITs, we obtained a final dataset consisting of 4,844 conversations. The statistics of the dataset are shown in Table 2.

We conducted additional filtering to further enhance the dataset quality. Utilizing gold answers obtained from the source OTT-QA dataset, we checked if the final answer appeared as a substring in Turker’s conversation. If it did, we auto-approved the conversation. For the remaining questions, we manually reviewed them. We approved conversations that had the correct answer but in a different format (e.g., September 1, 2021, instead of 9/1/21). In some cases, Turkers provided their own decomposition or their own ultimate question and decomposition, so they did not obtain the final answer provided by OTT-QA. In these cases, if the conversation was both accurate and had good quality, we accepted it.

The retrieval experiment is run for each $T_{0}$ of each conversation. Given the first question of the conversation $Q_{0}$, the model must predict the correct reference $R_{0}$. First questions discuss information that is either in a table or an intro paragraph; so the candidate space contains all intro paragraphs and tables in the dataset. The purpose of the retrieval experiment is to get a baseline of how well we are able to predict the table or page the subsequent conversation will be based upon, given the first query. The references that are utilized in the subsequent conversation are on the same page as the selected intro paragraph or table. For our baseline, we run the Okapi BM25 retriever Brown (2020) on the entire dataset over all candidates and first turn queries. BM25 is a standard document retrieval model that uses keyword-matching techniques to rank documents.

Previous work in dialogue systems focuses on the task of belief state tracking, which aims to determine the user’s goal or the current state of the conversation at each turn in the dialogue  Mrkšić et al. (2017); Ren et al. (2018). Inspired by work in belief state tracking, we propose the task of system state tracking in an information-seeking dialogue system. The task is framed similarly to belief state tracking, where a model attempts to classify the current state in the conversation at each turn. However, the “state” in our proposed task is modeled as a reference location from the current reference pool. As such, the task is formulated as using the information from the existing conversation and current question to determine the state of the conversation and choose which reference to utilize to create an answer. The reference types considered in this experiment are single cell, linked paragraph, inner table row, and multiple inner table rows. The implementation of system state tracking increases the interpretability and explainability of the system by determining the understanding of the user’s question and discovering the point in the conversation in which the model is incorrectly interpreting the user’s question. This, in turn, can help us understand the types of errors the model is prone to and allow us to work towards increasing the robustness of the model regarding these errors.

The system state tracking process is visualized in Figure 4. We perform system state tracking for all turns in each dialogue except the first turn. Given the history of the conversation $H_{i}$, we predict the correct reference $R_{i}$. $H_{i}$ consists of turns $T_{1}...T_{i-1}$, the current query $Q_{i}$, and the candidate references $RP_{i}$. Thus, the goal is to determine the correct reference $R_{i}$ at the specific turn in the dialogue, given the dialogue history. We utilize SentenceBERT Reimers and Gurevych (2019a) and TaPas Herzig et al. (2020) as baselines for the experiment.

We conduct experiments on dialogue response generation to look into the dataset’s expressivity for real-world dialogue scenarios. We fine-tuned a pre-trained DialoGPT model Zhang et al. (2020) by minimizing the negative log-likelihood with two input settings. $Q_{i}$, $A_{i}$, and $R_{i}$ are defined as the question, answer, and reference at the i-th turn, respectively. First, we only take the dialogue history as the input without knowledge content and predict the following natural language response. The format (DialoGPT-noR) is described as:

$$\{Q_{1},A_{1},...,Q_{i},A_{i},Q_{i+1}\}\mapsto A_{i+1}$$

(2)

Second, we flatten the references and concatenate the dialogue history as the input and predict the following natural language response. The references are flattened in the process seen in Figure 6. The format (DialoGPT) is:

$$\{R_{1},Q_{1},A_{1},...,R_{i+1},Q_{i+1}\}\mapsto A_{i+1}$$

(3)

The two settings enable us to validate how much information the references provide for response construction.

As retrieval is the first step in the information-seeking dialogue pipeline, we need to ensure that information from the correct Wikipedia page is retrieved to determine whether the first question and any following questions will be answerable. We evaluate our retrieval model with MRR@10 (Mean Reciprocal Rank @10). Table 3 shows our results, where BM25 achieves an MRR@10 score of 0.427 for retrieving the correct candidate.

We adopted SacreBLEU Post (2018) and BERTscore Zhang et al. (2019) as the automatic evaluation metrics. As shown in Table 5, concatenating references can consistently improve both metrics and the collected references are necessary for generating dialogue. It can be seen that differences are more noticeable for SacreBLEU as opposed to BERTscore. This is due to the naturally similar outputs of BERTscore, where the ranking of the scores is a more reliable view of the metric.

We conduct further error analysis and find three main types of errors as listed in Table 6: incoherent, non-fluent, and unfaithful. As shown in Table 6, the generated response “Alanis Nadine Morissette is a Canadian-American singer, songwriter, and actress.” is not an appropriate response to the question. In this case, the generated response is incoherent based on the dialogue. Sometimes the response has the correct information, but it is not a fluent sentence. One example is the generated statement “Yes, they performed to win the song La det swinge”. The final primary error type is that the generated response may be unfaithful to the perceived knowledge. For example, given a paragraph mentioning several years and events in history, the generated response mentions “1998”, while the answer should be “2005”.

In addition, we conduct a human evaluation. We randomly sample 200 test samples containing previous conversation histories, human-written answers, and machine-generated answers from DialoGPT. For each sample, we have two Turkers provide ratings. We ask the Turker to evaluate the machine-generated response on three criteria: coherence, fluency, and informativeness from a scale of 1 to 5. Coherence measures how well the response is connected to the question and prior conversation history. Fluency measures the use of proper English. Informativeness measures how accurate the machine-generated response is against the human-provided ground truth response. We provide the average ratings for each model in Table 7. The model that utilizes the state tracking references achieves a better "informativeness" rating as it is able to utilize the extra information to provide a more correct response. It is notable however that the model with no references achieves better coherence and fluency scores. Thus, the human evaluation demonstrates the importance and challenge for models to provide both an accurate and articulate response.