AnyTOD: A Programmable Task-Oriented Dialog System

An overview of the AnyTOD system is presented in Fig. 1. We decompose AnyTOD into three steps, and describe each step in detail below:

1. 1.

   Schema and program construction: A designer constructs a schema for AnyTOD to characterize the ontology of a specific task, as well as a policy graph that declares the task logic.

2. 2.

   Zero-shot DST and AST: A LM performs zero-shot DST and AST with reference to the schema, without task-specific training.

3. 3.

   Program execution and NAP: The predicted states and action history are passed to the schema program, which upon execution recommends preferred system actions to the agent. These actions are resent to the LM, which predicts the final system action(s) conditioned on these recommendations, conversation history, and belief states.

To train AnyTOD, we construct STARv2, an updated version of STAR with new ground truth belief state and action annotations, supporting supervised training on DST and AST. These annotations were generated from few-shot training with D3ST (Zhao et al., 2022). We first train D3ST on the SGD dataset, then continue finetuning on a few hand-labeled conversations from STAR.²²24 conversations were labeled from each task in STAR. While not the focus of this paper, the labeling of STARv2 demonstrates the use of few-shot D3ST in labeling unlabeled conversations on new tasks/domains.

Further, STARv2 adds new natural language descriptions for actions in STAR schemas. Prior work on STAR (Mosig et al., 2020; Mehri and Eskenazi, 2021) leverages template utterances as schema descriptions, which we qualitatively found to not fully outline the complexity of actions e.g., the action user_weather_inform_city has a template utterance of just [CITY]. STARv2 provides user is informing city as a more natural action description. We show in Section 4.2 that these actions improve zero-shot AST.

STAR and STARv2: As described in Section 3.2, we upgrade the original STAR (Mehri and Eskenazi, 2021) dataset to STARv2. The dataset has 24 tasks across 13 domains, many tasks requiring the model to adhere to a novel policy, providing an important zero-shot AST and NAP benchmark.

ABCD Chen et al. (2021): The design of the ABCD dataset follows a realistic setup, in which an agent’s actions must be balanced between the customer’s expressed desires and the constraints set by task policies. It is thus a natural fit for the AnyTOD framework for both training and evaluation.

SGD Rastogi et al. (2020): SGD is another schema-guided dataset in which schema elements are provided with natural language descriptions to facilitate task transfer. It contains 45 domains and was generated via simulation. Thus, the agent actions and responses follow pre-defined task logic.

MultiWOZ Budzianowski et al. (2018b): MultiWOZ is the standard dataset for benchmarking TOD models. It contains 7 domains and was generated through Wizard-of-Oz (Kelley, 1984) data collection, leading to natural conversations.

Table 1 shows AnyTOD results on the STARv2 dataset on the full-shot and zero-shot domain transfer settings, with both “happy” and “unhappy” conversations. In full-shot, models train on 80% of conversations across all tasks, and evaluate on the remaining 20%. The zero-shot domain setting is a leave-one-out cross-validation across the STARv2 dataset’s 13 domains, evaluating quality on an unseen schema in a completely novel domain. The following metrics are used in our report: joint goal accuracy (JGA) to measure DST, user action F1 (UaF1) to measure AST, system action F1 (SaF1) to measure NAP, and response BLEU.⁴⁴4See Section A.4 for details on calculating these metrics.

Each STAR task schema defines the intended dialog policy by providing a policy graph, where nodes describe conversation actions, and edges connect subsequent actions. An AnyTOD program (Figure A.2) is implemented to recommend next actions with respect to this policy graph.

Two baselines are used for comparison: BERT+S (Mosig et al., 2020) and SAM (Mehri and Eskenazi, 2021), both of which add a policy graph following module for zero-shot transfer to unseen schema. Note that, though these models were trained on the original STAR data, their SaF1 results are directly comparable to AnyTOD trained on STARv2 on NAP (SaF1), as these ground truth labels were left untouched. However, AnyTOD has additional training supervision on AST and DST due to STARv2’s new annotations. For a fair comparison with SAM, we also report results on SAM-User, a modified version of SAM trained on STARv2 that also includes supervised training on user annotations.⁵⁵5See Section A.5 for implementation details. Note that both BERT+S and SAM are based on BERT-base (110M parameters), comparable to T5 base (220M parameters).

Main Result The primary results for AnyTOD base/xxl are given in Table 1. For conciseness, we shorten AnyTOD to AT. As an ablation, we also report results with AT-norec, which removes the policy graph guidance from the AnyTOD method by recommending no system actions. In the full-shot setting, both AnyTOD and -norec, along with reported baselines, achieve very high SaF1. This is due to direct supervised training on NAP removing the need for program guidance. However, we see a huge gap between AnyTOD and -norec in the zero-shot setting; the guidance from the program becomes necessary — we see 60.6 vs. 55.8 SaF1 at base, and 68.0 vs. 62.3 SaF1 at xxl. Moreover, AnyTOD xxl has zero-shot performance comparable to that of full-shot, with 75.4 SaF1 at xxl.

Effect of Natural Language Descriptions As mentioned in Section 3.2, STARv2 provides new natural language descriptions that better characterize the actions within STAR. Our main result AT base/xxl takes advantage of these new descriptions, but to see the impact of these descriptions, we train AT-tmpl on the original template utterances. On base we see little difference between descriptions and templates, but a sizeable improvement in using descriptions appears on xxl, with a larger LM that is better at NLU. This evidences that more intuitive natural language descriptions help AnyTOD understand task semantics better and perform zero-shot transfer.

AnyTOD vs. baselines To compare against available results on STARv2, we compare AT-tmpl base against SAM-User. Both results use template responses provided by STAR, and additionally trained with the new DST and AST annotations in STARv2. However, we see far stronger performance with AnyTOD than with SAM or SAM-User, due to the flexibility provided by the program execution ability demonstrated by AnyTOD, and enabled by supervised training on DST and AST. SAM is not suited to use these contextual signals, likely due to no attention between schema elements and conversation and a rigid classification architecture unsuitable for multiple losses.

Multitask Training with SGD To demonstrate further robustness for AnyTOD, we also report AnyTOD-sgd, which jointly trains with SGD as a multitask training dataset. SGD includes a large number of tasks, each defined by a schema with highly diverse parameters and actions. The -sgd results in Table 1 show that at base, SGD multitask training improves both DST ($61.9\to 66.1$ JGA), AST ($72.1\to 74.3$ UaF1), and by extension NAP ($60.6\to 61.3$ SaF1). A similar but smaller improvement is seen on xxl, suggesting that the larger LM may not need more diverse training owing to its better language understanding.

Complex Program Logic STARv2 is also a good testbed for complex zero-shot task adaptation, as it includes some tasks which are more complex than simple policy-graph following, specifically the bank, trivia, and trip domains. For instance, the trivia task requires the agent to ask the user a trivia question and extract their answer. Different system actions must be taken by the agent depending on whether or not the user’s answer is correct. This logic is not captured by the provided policy graph alone, requiring more complex logic. AnyTOD is suitable for this problem, as we need only to construct a program implementing this logic. These programs are shown in Section A.1.

We report results with these programs in Table 1 under the -prog name. There is a clear win on zero-shot domain SaF1 when averaged over all domains, with a very high 70.7 SaF1 on -prog+sgd xxl, narrowing the gap with the full-shot 75.4 SaF1. When examining the complex tasks tasks individually (Table 1(c)), the win on NAP is even more apparent. The only exception is AT xxl on trivia, which has little difference with or without the program. In general however, the guidance provided by this specialized program is necessary for higher-level logic in the dialog policy, since the policy graph does not specify enough information to approach the task in zero-shot.

We conduct similar experiments on Action State Tracking (AST) (metric: joint action accuracy or JAA) on ABCD (Chen et al., 2021) and DST and NAP (metrics: JGA and SaF1 respectively) on SGD (Rastogi et al., 2020) datasets.

ABCD contains 10 flows, each describing the business logic for handling a customer request, which are relatively similar to each other. We report full-shot results by training and evaluating on all flows, and zero-shot results where the model is trained on one randomly sampled flow and evaluated on all other nine flows. The SGD test set consists of 21 services, 15 of these not seen during training. The dataset is generated via simulation with a generalized policy graph (shared across all services) encoding dialog act transitions. The per-service policy graphs are then constructed by inserting intents and slots and, as a result, end up similar.

Tables 2 and 4 and show AnyTOD results on SGD and ABCD respectively. For both datasets on both full-shot and zero-shot setups we generally see an improvement on action prediction using policy guidance, achieving state-of-the-art results for ABCD. However, the gain is not as large as STARv2, as the task policies are not as diverse. Even without explicit policy guidance, features from different tasks in ABCD/SGD can transfer to each other. Notably, policy guidance helps more on the one-flow setup for ABCD and unseen services for SGD, further establishing the efficacy of policy guidance on unseen setups, even if related.

To demonstrate the generalizability of the AnyTOD system, we demonstrate zero-shot transfer results on the end-to-end MultiWOZ 2.2 (Zang et al., 2020) benchmark, a popular dataset for TOD research. In this case, AnyTOD-xxl is trained on the SGD dataset, and then evaluated on MultiWOZ in zero-shot with a small policy program (Section A.6). Responses from AnyTOD were constructed using the template utterance approach from Kale and Rastogi (2020). We compare against SOLOIST (Peng et al., 2020) and Mars (Sun et al., 2022), two end-to-end TOD models directly trained on MultiWOZ with supervision. Results are shown in Table 5, with metrics reported by the MultiWOZ eval script (Nekvinda and Dusek, 2021). Although no training examples from MultiWOZ was used at all, AnyTOD demonstrates strong JGA, Inform, and Success comparable to results that do train on MultiWOZ. Note that since we applied templates for response generation, we do not consider BLEU to be important, as the responses are very different from ground truth labels.

To see how impactful the recommendations provided by the policy graph are, we reevaluate already finetuned AnyTOD models on the STARv2 zero-shot domain setting, but with changes to the program recommendations during eval. First, to see how dependent AnyTOD is on policy graph guidance, we modify the graph to output no recommendations (denoted as 0rec), forcing the model to do NAP only using the conversation, belief state, and action history. Secondly, we modify the graph to output deliberately bad recommendations (denoted as badrec), intended to trick the model into choosing an incorrect system action. This was done by randomly sampling 1-3 system actions other than the ground truth action.

The major drops in SaF1 for both setups shown in Table 4 confirm that the model, while able to predict actions without it, does consider the policy guidance heavily. Notably, 75% and 83% of correct predictions for 0rec and badrec are actions common to all tasks e.g., hello or query.

We conduct a similar “policy corruption” experiment on ABCD (Table 6), in which policy graphs for evaluation tasks have a 0%, 40%, and 80% chance of being replaced by graphs from incorrect flows during evaluation. We see a consistent quality drop with increasing probability of corruption for both base and xxl.

We also analyze AnyTOD errors on STARv2. We classify all incorrect NAPs into three possible error categories: (1) System action error: the program recommends the correct system action, but this was not chosen by the LM, (2) Policy graph error: the predicted belief state and action history are correct, but the program’s execution of the policy graph does not recommend the expected system action, and (3) State tracking error: the predicted belief states and action history are incorrect, which leads to incorrect recommendations from the policy graph. Results are shown in Figure 2. In general, we see that the benefit to scaling the LM from base to xxl comes from improvements to state and action tracking, which aligns with better DST and AST results on xxl as in Table 1.