SF-DST: Few-Shot Self-Feeding Reading Comprehension Dialogue State Tracking with Auxiliary Task

Conversation $C$ for time step $t$ is denoted as $C_{t}=\{x_{1},y_{1},...,y_{t-1},x_{t}\}$, where $x_{t}$ means user utterance and $y_{t}$ means system utterance. The belief state $B_{t}$ at turn ${t}$ is composed of slot $s\in S$ and value $v\in V$. $V$ includes don’t care and not mentioned values and notation $v_{s}$ means value for slot $s$. Question set ${Q}$ consists of ${q_{s}}$ which is predefined before training (e.g., $s$: attraction.name, ${q_{s}}$: What is the attraction name?). Auxiliary set ${A}$ consists of ${a_{s}}$, and ’Are they talking about [slot]?’ is the inquiry form. ${a_{s}}$ is also predefined before training (e.g., $s$: attraction.name, ${a_{s}}$: Are they talking about attraction name?).

SF-DST has text-to-text structure for generative, ontology-free DST (Figure 2). The input value consists of dialogue history $C_{t}$, corresponding question for slot $q_{s}$, and previous belief state $B_{t-1}$. The model answers the question for all slots at each dialogue turn $t$, and the predicted answers $v_{s}$ become $B_{t}$. We did not use a gold belief state as input in both training and inference time; instead, we designed a self-feeding belief state method in which the predicted belief state from the previous turn becomes the current turn’s input belief state. We separate user and system utterances by using [user] and [sys], and add index words Context, Question and Belief to distinguish each input part:

$$v_{s}=seq2seq(C_{t},q_{s},B_{t-1}).$$

(1)

We use negative log-likelihood as a loss function given $C_{t}$, $q_{s}$ and $B_{t-1}$ as

$$L_{belief}=-\sum_{i=1}^{n}\log p(v_{i}|C_{t},q_{i},B_{t-1}),$$

(2)

where $n$ denotes the total number of slots.

Some extractive DSTs use a two-step system. These systems first classify whether the slot is mentioned in dialogue, and if classified as mentioned, then finds the answer span from dialogues [9, 10, 11, 12]. This classification module is called slot-gate. By splitting the DST task into two models, this strategy can lower the strain on each model. However, this cascading approach risks error propagation and requires a relatively long inference time. Instead of a cascading strategy, we directly answered a question and trained a slot-gate task as an auxiliary task. The auxiliary task is trained as a question-answering form (Figure  2) and ’Are they talking about [slot]?’ is the inquiry format. The answer is Yes if the slot value is in belief state $B_{t}$, and not mentioned otherwise; i.e., the main DST question aims to generate a specific value, whereas the auxiliary question aims to classify the slot’s mention in the dialogue. The auxiliary task uses the context $C_{t}$, previous belief state $B_{t-1}$ and auxiliary question $a_{s}$ as input which has the same form as (1) except the slot question, and uses loss function (2). To train the auxiliary task with the main DST task, our model uses a joint loss function with hyperparameter $a$

$$L=L_{belief}+aL_{aux}.$$

(3)

We set $a$ empirically to 0.7.

We evaluated our model in a commonly-used supervised setting to compare with other DST models. In comparison, we included few/zero-shot DST baselines—TRADE [13], STARC [2], DSTQA [14], GPT2QA¹¹1The paper did not named the model. We assign temporary name to simplify comparison. [3]—and other general DST models, including DS-DST [15], SST-2 [16], TripPy [11], and FPDSC [17]. In addition to JGA, we also report ontology usage and answer prediction type. We divide answer predict type into classification (C), span prediction (S), and generative (G) following [3]. SF-DST achieved the highest accuracy by a wide margin compared to other few/zero-shot DST (Table 1). Our accuracy was lower than the best score of the classification-type method. However, these models need fixed ontology and find answers by classifying the value in ontology [2, 14, 15, 16, 17]. Fixed ontology is hard to obtain in the real world and cannot adapt to frequently changing values. In contrast, our model is ontology-free and generates the answer, which is competitive in the real world.

We perform an ablation study to investigate which component contributes to accuracy in a few-shot environment (10%). We observe that both self-feeding belief state and auxiliary task are essential to increase the accuracy (Table 4). Additionally, we trained the model with the gold belief state (+ Gold belief state in table) instead of self-feeding belief state. The JGA showed a significant decrease (38.55%) compared to training with a self-feeding belief state (44.60%). When the gold belief state is given during the training, the model depends on the belief state rather than the conversation. This causes the performance degrades in the inference stage, where the gold belief state cannot be given.

Although accurate DST requires an understanding of the entire dialogue, it is challenging when the dialogue has many turns. In this experiment, we analyzed the effect of the belief state according to the conversation length in a multi-turn circumstance. We separated the dialogue into three classes by the number of previous turns: short-length dialogue (one to three turns), medium-length dialogue (four to six turns), and long-length dialogue (seven or more turns). We trained the model with and without belief state in a few-shot setting (10%) and used the average of turn JGA. Our self-feeding belief state improved both short-length dialogue and medium-length dialogue (Table 5). This means that the belief state, which summarizes previous conversation information, helped the model to understand the multi-turn conversation. However, the JGA decreased when the dialogue was extended (more than seven turns). As the dialogue progressed, the probability of error propagating from the previous belief state increased, so the accuracy dropped. Therefore, finding a self-feeding method that reduces error propagation is a worthy future goal.

To examine the effect of the slot-gate auxiliary task, we classified the errors as Wrong, Ignore, and Spurious [2, 3]. Wrong means that the model correctly predicts the existence of the answer but predicts the wrong value. Ignore means that the answer exists, but the model ignores it. Spurious means that the answer is not mentioned, but the model predicts some value. We trained our model with and without auxiliary tasks at various training sizes and calculated the changed error and JGA rate by adapting the auxiliary task (Table 6). Overall, JGA was improved in all data settings. This result indicates that the auxiliary task helped to find accurate answers. In the case of error type, ignore, and spurious errors decreased; this result means that the auxiliary task was helpful to classify whether a slot is mentioned in the dialogue. However, the wrong type error grows in all settings. Adopting the auxiliary task increases the number of attempts to find the answer in dialogue when the answer exists, and this causes the growth of the wrong type error. Future work should find an auxiliary task that decreases all types of errors.

Finding the proper answer becomes increasingly challenging when the dialogue does not include an exact match. For example, assume the user said, ”I want to find a place to see a movie.” In that situation, even if the attraction type is not explicitly given, the model should infer the attraction type as theater. This implicit answer circumstance is common in the real world, and we experimented to determine whether our auxiliary question assists in such conditions. We chose ten slots²²2train.day, restaurant.area, hotel.star, attraction.area, hotel.stay, hotel.area, restaurant.day, hotel.people, hotel.day, restaurant.pricerange with a high probability of exact matching answers (explicit slot) and ten slots³³3hotel.type, hotel.internet, hotel.parking, taxi.leaveat, attraction.name, taxi.departure, attraction.type, train.leaveat, taxi.destination, hotel.name with a low probability of exact matching answers (implicit slot) [2]. In the case of explicit slots, 99.12 % of the answers were exactly found in dialogue, compared to 70.34% in implicit slots. We experimented in a few-shot setting and used the JGA of targeted slots. Our auxiliary task improved JGA in all training data set and slot types (Table 7). Asking whether the slot is mentioned helps find an answer even if there is no exact match exists.