TicketTalk: Toward human-level performance with end-to-end, transaction-based dialog systems

Over the past few years the NLP community has responded to the lack of dialog data with larger, publicly released task-oriented datasets spanning multiple domains Wu et al. (2020); Budzianowski and Vulić (2019). This underscores the crucial role data plays in any approach to task-based dialog systems. MultiWOZ Budzianowski et al. (2018) consists of 10,420 dialogs in multiple domains and has become a popular benchmarking corpus for state tracking. It has also undergone a series of subsequent refinements. MSR-E2E, featured in the Microsoft dialog challenge Li et al. (2018), has 10,087 dialogues in three domains, movie-ticket booking, restaurant reservation, and taxi booking. Byrne et al. (2019) offers 13,215 dialogs in six domains and has been updated with a second installment, Taskmaster-2 Byrne et al. (2020), which adds 17,289 more dialogs totalling over 30,000. The Schema Guided Dialogue dataset Rastogi et al. (2020) has 22,825 dialogs in multiple domains. MetaLWOZ Lee et al. (2019) has 37,884 dialogs in 227 domains and is aimed at helping models more accurately predict user responses in new domains. Both Schema and MetaLWOZ are used in DSTC8 Kim et.al (2019). In addition to these, Serban et al. (2018) provides a thorough survey of dialog corpora released in previous years.

In contrast to the end-to-end ¹¹1The term “end-to-end” is sometimes also used when describing parts of modular systems Li et al. (2017); Wen et al. (2017) but it is fundamentally different from the single text-to-text transformer model approach we present here. approach, traditional, modular strategies employ a division of labor among the components, e.g. understanding, state tracking, dialog policy, generation, etc., which are either largely hand-crafted or derived from training individual models on labeled datasets Wen et al. (2017); Young et al. (2013). This architecture is inherently more complex than the single-model end-to-end strategy we propose and can require significantly more design and engineering. Moreover, since each module requires its own supervised training dataset, it is harder to apply to different domains Serban et al. (2015a).

However, the separation of functions makes the modular approach more transparent and in some respects easier to debug. It has also been considered by some to be better equipped to interact with external APIs Sukhbaatar et al. (2015); Wen et al. (2017) and therefore might be better suited for task-based dialogs. As mentioned above, we show that our single model-based approach can accurately generate both the appropriate response as well as predict the correct API call at the right time.

The TicketTalk movie ticketing dataset was created using the self-dialog collection method Krause et al. (2017); Moghe et al. (2018); Byrne et al. (2019) where a crowd-sourced worker writes both sides of the dialog (i.e. both customer and ticketing agent turns) based on a particular scenario and set of instructions. Following the annotation strategy used for Taskmaster-1 Byrne et al. (2019)), we limit labels to basic entities and events (i.e. API calls).

The rationale for limiting dialogs to a single domain (movie ticketing) is based on our hypothesis that human-level performance in terms of both response generation and API call prediction for a particular task requires larger (i.e. 10,000+), more diverse datasets than are currently available. In other words, carefully curated, annotated datasets that cover all the idiosyncrasies of a single task or transaction are a key factor in model performance. Concern about the cost and efficiency of creating these larger corpora has led some researchers to look for approaches that alleviate dependencies on annotated data Budzianowski and Vulić (2019); Wen et al. (2017). However, significant time and expense can be saved when assembling these corpora by simplifying the collection and annotation procedures. In addition, little to no training is required for workers to be able to perform consistently well.

Using self-dialogs (where a worker creates the whole conversation, both user and agent turns) facilitates building large and linguistically rich datasets since it is both simple and cost effective, and allows users to draw on their lifetime of conversational experiences. This in turn ensures the model can handle the wide range of human conversational behaviors that emerge in natural dialog. For this project we extended the self-dialog to include over three dozen sets of user instructions to generate a wider variety of conversations, from open-ended prompts to more specific instructions that require specific types of exchanges. For example, one set simply instructs workers to “write the transcription of a conversation” in which a person makes a successful ticket transaction with a booking agent. This allows dialog creators to express their unique view of what a typical movie ticketing transaction would be, structuring each conversation how they see fit. They are also instructed to find real values for required details (i.e. slots) such as time, date, theater, movie, etc. using a movie or theater site of their choice for a specific location. This ensures the dataset has a large and diverse KB. In contrast, the more restrictive sets of instructions focus on specific sub-dialogs for error handling, changing a detail, entity resolution, and the like. In such cases we often provide a limited KB with one or more values for all the details so the worker can focus on the primary task of creating a realistic set of exchanges for this type of interaction. In a third type of scenario, the conversation is partially completed and the user’s task is focused on a very specific part of the exchange. This allows us to “fill holes” in the data quickly and cost effectively. That is, we can create large numbers of short, conversational examples that the model does not handle adequately and then retrain for better results.

Dialog data annotation can be complex and time consuming even for trained linguists as it typically involves carefully and consistently labeling dialog states, user intents, and dialog acts, among other possible labels Henderson et al. (2013); Wen et al. (2017); Budzianowski et al. (2018). The API-targeted approach is far more straightforward since only basic entities (e.g. name, time, number of tickets, theater, movie attributes, etc.) and API calls (e.g. to find theaters, movies, and showtimes, book tickets, etc.) are labeled. The task is therefore easier to learn, faster to complete, and cheaper to run. Moreover, as we discuss below, it fits well with the text-to-text format we use in our approach to transaction-based dialog systems. The full annotation schema is included with the dataset release.

We implement a new approach to end-to-end dialog systems by combining natural language output and structured API calls into a unified text-to-text format where the input and output are always text strings. This allows us to leverage widely available, state of the art, general purpose text-to-text transformers as the foundation of our system. Specifically, we used the publicly available Text-To-Text Transfer Transformer (T5) Raffel et al. (2019) to train our models. The T5 framework was designed specifically to explore transfer learning techniques for NLP and includes pre-training on the Colossal Clean Crawled Corpus (C4), composed of hundreds of gigabytes of web-based English text Raffel et al. (2019). The original pre-training objective for the C4 corpus in the T5 framework was a denoising task, i.e. recovering missing words from the input. Since this type of task scales well to multiple downstream tasks, we used our custom inputs/targets from the TicketTalk dataset to represent an end-to-end task based dialog system and ultimately achieve positive results.

We use T5-Base Raffel et al. (2019) as our pre-trained model, which follows the transformer architecture Vaswani et al. (2017) and consists of 220M parameters. It was pre-trained on the large scale C4 dataset mentioned above for 1M steps with a span corruption objective. We fine-tune this model on the Taskmaster-3 dataset for 40000 steps with a constant learning rate of 0.001 using 16 TPU v3 chips. The batch size was set to 131,072 tokens per batch. The maximum input sequence length and output length were set to 1024 and 256 tokens respectively.

The goal of our model is to generate a text string that either serves as a verbal response to the user or that contains one or more API calls with the data required at the current stage of the conversation. Verbal responses come in two flavors: those that depend on a particular API call details and those that do not. For example, when an API is invoked to find theater names for a given movie and location, the details returned from the API call must be correctly incorporated into the system’s next response, e.g. “I found two theaters, AMC 20 and Century City 16.” In contrast, other verbal outputs, e.g. “What city do you plan to see the movie in?” are derived from the overall conversation history.

Given the required text-to-text format used in our approach, we identify the type and function of each string by converting the annotations to a set of tokens. As shown in Table 2, tokens identify the speaker, i.e. user vs. agent, the string type i.e. utterance vs. API call, and the details of each API call, both names as well as input parameters and values, and response parameters and values. We also tag the conversation “context” which separates the most recent turn from previous turns. Our token key is shown in Table 2.

The first step is to use tokens to represent the user and agent interactions, providing speaker information to the model by the use of ”$<$U$>$” and ”$<$A$>$”. We then convert any API invocations into their text equivalent using tokens for marking API names, argument types and values, i.e. ”$<$PN$>$”, ”$<$PAN$>$”, etc. The results of these two steps are shown in Table 3.

The next step is to create the model inputs and targets. We use the following algorithm to accomplish this:

1. 1.

   Initialize conversation context to an empty string.

2. 2.

   Iterate through the interactions and do the following:

   1. (a)

      If the sentence is a user utterance ($<$U$>$) or a program response($<$PR$>$), add it to the model input along with the conversation context (if present).

   2. (b)

      If the sentence is an agent utterance ($<$A$>$) or program invocation ($<$PN$>$), add it to the model target.

   3. (c)

      If both model input and target have been created, output the (input, target) pair and update the conversation context to reflect this.

   4. (d)

      Continue (2) to generate the next input, target pair.

Using the these rules, the model inputs and targets are generated as in Table 4.

Once the model has been trained on inputs and targets, we can use the system to accomplish tasks in the following manner:

1. 1.

   Obtain user utterance and format it by adding the speaker token.

2. 2.

   Provide the formatted utterance to the model.

3. 3.

   Obtain model prediction

   1. (a)

      If the model prediction contains the agent ($<$A$>$) token, format it and show it to the user.

      1. i.

         Update conversation context and start again from (1).

   2. (b)

      If the model prediction contains the program ($<$PN$>$) token:

      1. i.

         Extract program argument name ($<$PAN$>$) and value ($<$PAV$>$).

      2. ii.

         Issue the API call by providing it to the API adapter.

      3. iii.

         Format API results and provide it to the model along with the conversation context.

      4. iv.

         Start from (3).

This interaction lifecycle is illustrated in Figure 3.

When we detect an API call in the output, we invoke the API, retrieve the results, and embed the responses in the next model input. As shown in Figure 4, each API call predicted by the model typically contains a generic API name, such as ”find-movies”, or ”find-theaters”, and a list of key value pairs that detail the specific parameters to be used while invoking the API, as shown in Figure 4.

The API call, while structured, may still include pronouns or other co-referential phrases as input parameters. For example, the date parameter for an API call might contain the value “tonight”, and the location value might be “nearby”. The resolution of these entities happens outside the core interaction layer in what can be understood as the “API adapter” (and not the actual API itself). This not only helps simplify annotation, but also helps leverage existing solutions to these well defined problems. This separation of the API layer is also useful for encapsulating all API specific artifacts, like authentication tokens, endpoint addresses and data formatters. In this way, the end-to-end system is able to interact with the user to solicit details relevant to the task, generate API calls to fetch data from external knowledge sources, and use the responses provided by the API call to construct natural language responses.

In this section, we show how our end-to-end approach to transaction-based dialog systems produces verbal responses and predicts API calls with near human-level quality and accuracy. Through human qualitative evaluations, we show that two aspects in particular, dataset size and pre-training, significantly affect performance. Below we describe our evaluation methodology followed by a detailed discussion of the experiment results.

Dataset size and pre-training are key factors in creating models for end-to-end dialog systems. To understand the amount of data required for our approach, we trained four models, each on a different number of randomly selected subsets of the TicketTalk dataset, namely 5000, 7500, 10,000 and 21,000 dialogs. To measure the effect of transfer learning, we trained a second 10,000-dialog model without the T5 framework’s pre-training component, setting up an A-B comparison with the pre-trained model.

As mentioned earlier, our models generate three types of output: API calls, verbal responses based on the results of an API call, and “plain” verbal responses based on the conversation context (i.e. not dependent on a particular API call response). We set up a pair of evaluations for each type. The first evaluation asked human raters to evaluate the model’s output given a specific conversation history (i.e. context) while the second asked raters to evaluate the human’s response for the same set of contexts. Each experiment included 1000 context-response pairs of varying lengths, i.e. some conversation histories might have just one exchange (a user and agent turn) while others could have up to nine exchanges. We requested three ratings for each question distributed among a pool of about 900 paid raters for a total of 3000 data points per experiment. Table 5 and Table 6 below shows a sample context-response pair presented to human raters for each type of model output.

We use our “makes-sense” metric to evaluate the model-generated responses and API call predictions against the human standard. For verbal responses, we ask one question:

• •

  Does the agent’s next response make sense?

For negative answers, we give a list of reasons raters believe it does not make sense (i.e. off topic, repeated information, incorrect details, language mistakes, other). For API call predictions there are two questions:

1. 1.

   Do all the action types, their details, and their order make sense at this point in the conversation?

2. 2.

   Are there any actions that should be listed here but that are missing (either as additions or replacements)?

Again, raters are given options to choose for negative answers.

This offline evaluation strategy offers scalability and minimal rater training. However, an online, interactive evaluation infrastructure would allow us to evaluate the ability of the model to handle errors in its own output (from previous predictions) and its robustness while dealing with novel inputs. Future evaluation will be carried out on this new infrastructure.

Comparing the “makes-sense” scores for model-generated vs. human-generated responses, a clear pattern of improvement emerges based on dataset size. As shown in Table 7, when 5K and 7.5K dialogs are used for the training set, scores for model-generated responses lag behind the human-generated scores by up to 5.5%. At 10K dialogs, the response scores differ by less than 2% and model-generated API predictions outperform human labels by 2.5%. At 21K dialogs, model-generated responses improve to near human-level performance. The 21K model’s API call prediction fares better than human API labeling. As an automatic metric, we also provide the BLEU score generated for each model.

The effect of pre-training is also very clear. After training a fifth model, this time without the T5 framework’s pre-training feature, we see a huge drop in evaluation scores. As shown at the bottom of Table 7, we see a decrease of 30% in model performance for verbal responses and about a 25% drop in API call prediction accuracy.

Finally, the quality of the model’s prediction stays on par with human scores throughout the conversation as the context grows. Figure 5 shows how the model’s ”makes sense” score stay on the same path after each exchange.