NeuralWOZ: Learning to Collect Task-Oriented Dialogue
via Model-Based Simulation

Wizard-of-Oz (WOZ) is a widely used approach for constructing dialogue data Henderson et al. (2014a, b); El Asri et al. (2017); Eric and Manning (2017); Budzianowski et al. (2018). It works by facilitating a role play between two people. “User” utilizes a goal instruction that describes the context of the task and details of request and “system” has access to a knowledge base, and query results from the knowledge base. They take turns to converse, while the user makes requests one by one following the instructions, the system responds according to the knowledge base, and labels user’s utterances.

Other studies on dialogue datasets use the user simulator-based data collection approaches Schatzmann et al. (2007); Li et al. (2017); Bordes et al. (2017); Shah et al. (2018); Zhao and Eskenazi (2018); Shah et al. (2018); Campagna et al. (2020). They define domain schema, rules, and dialogue templates to simulate user behavior under certain goals. The ingredients to the simulation are designed by developers and the dialogues are realized by predefined mapping rules or paraphrasing by crowdworkers.

If a training corpus for the target domain exists, neural models that synthetically generates dialogues can augment the training corpus Hou et al. (2018); Yoo et al. (2019). For example, Yoo et al. (2020) introduce Variational Hierarchical Dialog Autoencoder (VHDA), where hierarchical latent variables exist for speaker identity, user’s request, dialog state, and utterance. They show the effectiveness of their model on single-domain DST tasks. SimulatedChat Mohapatra et al. (2020) also uses goal instruction for dialogue augmentation. Although it does not solve zero-shot learning task with domain expansion in mind, we run auxiliary experiments to compare with NeuralWOZ, and the results are in the Appendix D.

In zero-shot domain transfer tasks, there is no data for target domain, but there exists plenty of data for other domains similar to target domain. Solving the problem of domain expansion of dialogue systems can be quite naturally reducted to solving zero-shot domain transfer. Wu et al. (2019) conduct a landmark study on the zero-shot DST. They suggest a model, Transferable Dialogue State Generator (TRADE), which is robust to a new domain where few or no training data for the domain exists. Kumar et al. (2020) and Li et al. (2021) follow the same experimental setup, and we also compare NeuralWOZ in the same experiment setup. Abstract Transaction Dialogue Model (ATDM) Campagna et al. (2020), another method for synthesizing dialogue data, is another baseline for zero-shot domain transfer tasks we adopt. They use rules, abstract state transition, and templates to synthesize the dialogue, which is then fed into a model-based zero-shot learner. They achieved state-of-the-art in the task using the synthetic data on SUMBT Lee et al. (2019), a pretrained BERT Devlin et al. (2019) based DST model.

Collector is a sequence-to-sequence model, which takes a goal instruction $G$ and API call results $A$ as the input and generates dialogue $D_{T}$. The generated dialogue $D_{T}=(r_{1},u_{1},...,r_{T},u_{T})$ is the sequence of system response $r$ and user utterance $u$. They are represented by $N$ tokens $(w_{1},...,w_{N})$²²2Following Hosseini-Asl et al. (2020), we also utilize role-specific special tokens <system> and <user> for the $r$ and $u$ respectively..

$$p(D_{T}|G,A)=\prod_{i=1}^{N}p(w_{i}|w_{<i},G,A)$$

We denote the input of Collector as $\texttt{<s>}\oplus G\oplus\texttt{</s>}\oplus A$, where the $\oplus$ is concatenate operation. The <s> and </s> are special tokens to indicate start and seperator respectively. The tokenized natural language description of $G$ is directly used as the tokens. The $A$ takes concatenation of each $a_{i}$ ($a_{1}\oplus\cdots\oplus a_{|A|}$)³³3we limit the $|A|$ to a maximum 3. For each $a_{i}$, we flatten the result to the token sequence, $\texttt{<domain>}\oplus domain_{a_{i}}\oplus\texttt{<slot>}\oplus S_{1}^{a_{i}}\oplus V_{1}^{a_{i}}\oplus\cdots\oplus\texttt{<slot>}\oplus S_{|C^{a_{i}}|}^{a_{i}}\oplus V_{|C^{a_{i}}|}^{a_{i}}$. The <domain> and <slot> are other special tokens as separators. The objective function of Collector is

$$\mathcal{L}_{C}=-\frac{1}{M_{C}}\sum_{j=1}^{M_{C}}\sum_{i=1}^{N_{j}}\log p(w_{i}^{j}|w_{<i}^{j},G^{j},A^{j}).$$

Our Collector model uses the transformer architecture Vaswani et al. (2017) initialized with pretrained BART Lewis et al. (2020). Collector is trained using negative log-likelihood loss, where $M_{C}$ is the number of training dataset for Collector and $N_{j}$ is target length of the $j$-th instance. Following Lewis et al. (2020), label smoothing is used during the training with the smoothing parameter of 0.1.

We formulate labeling as a multiple-choice problem. Specifically, Labeler takes a dialogue context $D_{t}=(r_{1},u_{1},...,r_{t},u_{t})$, question $q$, and a set of answer options $O=\{o_{1},o_{2},...,o_{|O|}\}$, and selects one answer $\tilde{o}\in O$. Labeler encodes the inputs for each $o_{i}$ separately, and $s_{o_{i}}\in\mathbb{R}^{1}$ is the corresponding logit score from the encoding. Finally, the logit score is normalized via softmax function over the answer option set $O$.

		$\displaystyle p(o_{i}|D_{t},q,O)=\frac{\exp(s_{o_{i}})}{\sum_{j}^{|O|}\exp(s_{o_{j}})},$
		$\displaystyle s_{o_{i}}=Labeler(D_{t},q,o_{i}),\forall i.$

The input of Labeler is a concatenation of $D_{t}$, $q$, and $o_{i}$, $\texttt{<s>}\oplus D_{t}\oplus\texttt{</s>}\oplus q\oplus\texttt{</s>}\oplus o_{i}\oplus\texttt{</s>}$, with special tokens. For labeling dialogue states to $D_{t}$, we use the slot description for each corresponding slot type, $S_{i}$, as the question, for example, “what is area or place of hotel?” for “hotel-area” in Figure 2. We populate corresponding answer options $O_{S_{i}}=\{V_{j}|(S_{j},V_{j})\in C,S_{j}=S_{i}\}$ from the state candidate set $C$. There are two special values, $Dontcare$ to indicate the user has no preference and $None$ to indicate the user is yet to specify a value for this slot Henderson et al. (2014a); Budzianowski et al. (2018). We include these values in the $O_{S_{i}}$.

For labeling the active domain of $D_{t}$, which is the domain at $t$-th turn of $D_{t}$, we define domain question, for example “what is the domain or topic of current turn?”, for $q$ and use predefined domain set $O_{domain}$ as answer options. In MultiWOZ, $O_{domain}$ = {“Attraction”, “Hotel”, “Restaurant”, “Taxi”, “Train”}.

Our Labeler model employs a pretrained RoBERTa model Liu et al. (2019) as the initial weight. Dialogue state and domain labeling are trained jointly based on the multiple choice setting. Preliminary result shows that the imbalanced class problem is significant in the dialogue state labels. Most of the ground-truth answers is $None$ given question⁴⁴4The number of $None$ in the training data is about 10 times more than the number of others. Therefore, we revise the negative log-likelihood objective to weight other (not-$None$) answers by multiplying a constant $\beta$ to the log-likelihood when the answer of training instance is not $None$. The objective function of Labeler is

		$\displaystyle\mathcal{L}_{L}=-\frac{1}{M_{L}}\sum_{j=1}^{M_{L}}\sum_{t=1}^{T}\sum_{i=1}^{N_{q}}\mathcal{L}_{t,i}^{j}$
		$\displaystyle\mathcal{L}_{t,i}^{j}=\begin{cases}\beta\log p(\tilde{o}_{t,i}^{j}|D_{t}^{j},q_{i}^{j},O_{i}^{j}),&\text{if }\tilde{o}_{t,i}^{j}\neq\text{$None$}\\ \log p(\tilde{o}_{t,i}^{j}|D_{t}^{j},q_{i}^{j},O_{i}^{j}),&\text{otherwise}\end{cases}$

, where $\tilde{o}_{t,i}^{j}$ denotes the answer of $i$-th question for $j$-th training dialogue at turn $t$, the $N_{q}$ is the number of questions, and $M_{L}$ is the number of training dialogues for Labeler. We empirically set $\beta$ to a constant $5$.

We first define goal template $\mathcal{G}$.⁵⁵5In Budzianowski et al. (2018), they also use templates like ours when allocating goal instructions to the user in the Wizard-of-Oz setup. $\mathcal{G}$ is a delexicalized version of $G$ by changing each value $V_{i}^{G}$ expressed on the instruction to its slot $S_{i}^{G}$. For example, the “expensive” and “british” of goal instruction in Figure 1 are replaced with “restaurant-pricerange” and “restaurant-food”, respectively. As a result, domain transitions in $\mathcal{G}$ becomes convenient.

First, $\mathcal{G}$ is sampled from a pre-defined set of goal template. API call results $\mathcal{A}$, which correspond to domain transitions in $\mathcal{G}$, are randomly selected from the $KB$. Especially, we constrain the sampling space of $\mathcal{A}$ when the consecutive scenario among domains in $\mathcal{G}$ have shared slot values. For example, the sampled API call results for restaurant and hotel domain should share the value of “area” to support the following instruction “I am looking for a hotel nearby the restaurant”. $\mathcal{G}$ and $\mathcal{A}$ are aligned to become $G_{\mathcal{A}}$. In other words, each value for $S_{i}^{G}$ in $\mathcal{G}$ is assigned using the corresponding values in $\mathcal{A}$.⁶⁶6Booking-related slots, e.g., the number of people, time, day, and etc., are randomly sampled for their values since they are independent of the $A$. Then, Collector generates dialogue $\mathcal{D}$, of which the total turn number is $T$, given $G_{\mathcal{A}}$ and $\mathcal{A}$. More details are in Appendix A. Nucleus sampling Holtzman et al. (2020) is used for the generation.

We denote dialogue state and active domain at turn $t$ as $B_{t}$ and $domain_{t}$ respectively. The $B_{t}$, $\{(S_{j},V_{j,t})\mid 1\leq j\leq J\}$, has $J$ number of predefined slots and their values at turn $t$. It means Labeler is asked $J$ (from slot descriptions) + 1 (from domain question) questions regarding dialogue context $\mathcal{D}_{t}$ from Collector. Finally, the output of Labeler is a set of dialogue context, dialogue state, and active domain at turn $t$ triples $\{(\mathcal{D}_{1},B_{1},domain_{1}),...,(\mathcal{D}_{T},B_{T},domain_{T})\}$.

We use MultiWOZ 2.1 Eric et al. (2019) dataset⁷⁷7https://github.com/budzianowski/multiwoz for our experiments. It is one of the largest publicly available multi-domain dialogue data and it contains 7 domains related to travel (attraction, hotel, restaurant, taxi, train, police, hospital), including about 10,000 dialogues. The MultiWOZ data is created using WOZ so it includes goal instruction per each dialogue and domain-related knowledge base as well. We train our NeuralWOZ using the goal instructions and the knowledge bases first. Then we evaluate our method on dialogue state tracking with and without synthesized data from the NeuralWOZ using five domains (attraction, restaurant, hotel, taxi, train) in our baseline, and follow the same preprocessing steps of  Wu et al. (2019); Campagna et al. (2020).

We use the pretrained BART-Large Lewis et al. (2020) for Collector and RoBERTa-Base Liu et al. (2019) for Labeler. They share the same byte-level BPE vocab Sennrich et al. (2016) introduced by Radford et al. (2019). We train the pipelined models using Adam optimizer Kingma and Ba (2017) with learning rate 1e-5, warming up steps 1,000, and batch size 32. The number of training epoch is set to 30 and 10 for Collector and Labeler respectively.

For the training phase of Labeler, we use a state candidate set from ground truth dialogue states $B_{1:T}$ for each dialogue, not like the synthesizing phase where the options are obtained from goal instruction and API call results. We also evaluate the performance of Labeler itself like the training phase with validation data (Table 5). Before training Labeler on the MultiWOZ 2.1 dataset, we pretrain Labeler on DREAM⁸⁸8The DREAM is a multiple-choice question answering dataset in dialogue and includes about 84% of non-extractive answers. Sun et al. (2019) to boost Labeler’s performance. This is similar to coarse-tuning in Jin et al. (2019). The same hyper parameter setting is used for the pretraining.

For the zero-shot domain transfer task, we exclude dialogues which contains target domain from the training data for both Collector and Labeler. This means we train our pipelines for every target domain separately. We use the same seed data for training as  Campagna et al. (2020) did in the few-shot setting. All our implementations are conducted on NAVER Smart Machine Learning (NSML) platform Sung et al. (2017); Kim et al. (2018) using huggingface’s transformers library Wolf et al. (2020). The best performing models, Collector and Labeler, are selected by evaluation results from the validation set.

We synthesize 5,000 dialogues for every target domain for both zero-shot and few-shot experiments⁹⁹9In Campagna et al. (2020), the average number of synthesized dialogue over domains is 10,140., and 1,000 dialogues for full data augmentation. For zero-shot experiment, since the training data are unavailable for a target domain, we only use goal templates that contain the target domain scenario in the validation set similar to Campagna et al. (2020). We use nucleus sampling in Collector with parameters top_p ratio in the range $\{0.92,0.98\}$ and temperature in the range $\{0.7,0.9,1.0\}$. It takes about two hours to synthesize 5,000 dialogues using one V100 GPU. More statistics is in Appendix B.

We compare NeuralWOZ with baseline methods both zero-shot learning and data augmentation using MultiWOZ 2.1 in our experiments. We use a baseline zero-shot learning scheme which does not use synthetic data Wu et al. (2019). For data augmentation, we use ATDM and VHDA.

ATDM refers to a rule-based synthetic data augmentation method for zero-shot learning suggested by Campagna et al. (2020). It defines rules including state transitions and templates for simulating dialogues and creates about 10,000 synthetic dialogues per five domains in the MultiWOZ dataset. Campagna et al. (2020) feed the synthetic dialogues into zero-shot learner models to perform zero-shot transfer task for dialogue state tracking. We also employ TRADE Wu et al. (2019) and SUMBT Lee et al. (2019) as baseline zero-shot learners for fair comparisons with the ATDM.

VHDA refers to model-based generation method using hierarchical variational autoencoder Yoo et al. (2020). It generates dialogues incorporating information of speaker, goal of the speaker, turn-level dialogue acts, and utterance sequentially. Yoo et al. (2020) augment about 1,000 dialogues for restaurant and hotel domains in the MultiWOZ dataset. For a fair comparison, we use TRADE as the baseline model for the full data augmentation experiments. Also, we compare ours with the VHDA on the single-domain augmentation setting following their report.

Our method achieves new state-of-the-art of zero-shot domain transfer learning for dialogue state tracking on the MultiWOZ 2.1 dataset (Table 1). Except for the hotel domain, the performance over all target domains is significantly better than the previous sota method. We discuss the lower performance in hotel domain in the analysis section. Following the work of Campagna et al. (2020), we also measure zero-shot coverage, which refers to the accuracy ratio between zero-shot learning over target domain, and fully trained model including the target domain. Our NeuralWOZ achieves 66.9% and 79.2% zero-shot coverage on TRADE and SUMBT, respectively, outperforming previous state-of-the-art, ATDM, which achieves 61.2% and 73.5%, respectively.

For full data augmentation, our synthesized data come from fully trained model including all five domains in this setting. Table 2 shows that our model still consistently outperforms in full data augmentation of multi-domain dialogue state tracking. Specifically, our NeuralWOZ performs 2.8% point better on the joint goal accuracy of TRADE than ATDM. Our augmentation improves the performance by a 1.6% point while ATDM degrades.

We also compare NeuralWOZ with VHDA, a previous model-based data augmentation method for dialogue state tracking Yoo et al. (2020). Since the VHDA only considers single-domain simulation, we use single-domain dialogue in hotel and restaurant domains for the evaluation. Table 3 shows that our method still performs better than the VHDA in this setting. NeuralWOZ has more than twice better joint goal accuracy gain than that of VHDA.

Table 4 shows the intrinsic evaluation results from two components (Collector and Labeler) of the NeuralWOZ on the validation set of MultiWOZ 2.1. We evaluate each component using perplexity for Collector and joint goal accuracy for Labeler, respectively. Note that the joint goal accuracy is achieved by using state candidate set, prepopulated as the multiple-choice options from the ground truth, $B_{1:T}$, as the training time of Labeler. It can be seen as using meta information since its purpose is accurate annotation but not the dialogue state tracking itself. We also report the results by excluding target domain from full dataset to simulate zero-shot environment. Surprisingly, synthesized data from ours performs effectively even though the annotation by Labeler is not perfect. We conduct further analysis, the responsibility of each model, in the following section.

Figure 3 shows the slot accuracy for each slot type in the hotel domain, which is the weakest domain from ours. Different from other four domains, only the hotel domain has two boolean type slots, “parking” and “internet”, which can have only “yes” or “no” as their value. Since they have abstract property for the tracking, Labeler’s labeling performance tends to be limited to this domain. However, it is noticeable that our accuracy of booking related slots (book stay, book people, book day) are much higher than the ATDM’s. Moreover, the model using synthetic data from the ATDM totally fails to track the “book stay” slot. In the synthesizing procedures of Campagna et al. (2020), they create the data with a simple substitution of a domain noun phrase when the two domains have similar slots. For example, “find me a restaurant in the city center” can be replaced with “find me a hotel in the city center” since the restaurant and hotel domains share “area” slot. We presume it is why they outperform over slots like “pricerange” and “area”.

We further investigate how our method is complementary with human-annotated data. Figure 4 illustrates our NeuralWOZ shows a consistent gain in the few-shot domain transfer setting. Unlike the performance with ATDM is saturated as few-shot ratio increases, the performance using our NeuralWOZ is improved continuously. We get about 5.8% point improvement from the case which does not use synthetic data when using 10% of human-annotated data for the target domain. It implies our method could be used more effectively with the human-annotated data in a real scenario.

We discover whether Collector and Labeler are more responsible for the quality of synthesizing. Table 5 shows ablation results where each model of NeuralWOZ is trained the data including or withholding the hotel domain. Except for the training data for each model, the pipelined models are trained and dialogues are synthesized in the same way. Then, we train TRADE model using the synthesized data and evaluate it on hotel domain like the zero-shot setting. The performance gain from Collector which is trained including the target domain is 4.3% point, whereas the gain from Labeler is only 0.8% point. It implies the generation quality from Collector is more responsible for the performance of the zero-shot learner than accurate annotation of Labeler.

Figure 5 is an qualitative example generated by NeuralWOZ. It shows the NeuralWOZ can generate an unseen movie domain which has a different schema from the traveling, the meta domain of the MultiWOZ dataset, even if it is trained on only the dataset. It is harder to generalize when the schema structure of the target domain is different from the source domain. Other examples can be found in Appendix C. We would like to extend the NeuralWOZ to more challenging expansion scenario like these in future work.

To show that our framework can be used for other dialogue tasks, we test our data augmentation method on end-to-end task in MultiWOZ 2.1. We describe the result in Appendix D with discussion. In full data setting, Our method achieves 17.46 BLUE, 75.1 Inform rate, 64.6 Success rate, and 87.31 Combine rate, showing performance gain using the synthetic data. Appendix D also includes the comparison and discussion on SimulatedChat Mohapatra et al. (2020).