A Low-Cost, Controllable and Interpretable Task-Oriented Chatbot: With Real-World After-Sale Services as Example

A dialogue action is regarded as a group of utterances with unique and identical semantic information. Inspired by Lv et al. (2021), we exploit a popular two-stage method to cluster utterances from large scale dialogues as follows.

Feature Extractor Sentence representation generated by pre-trained language models has been widely used as features for clustering. In this paper, we exploit ConSERT (Yan et al., 2021), which solves the collapse issue of BERT-derived sentence representations by contrastive learning, as feature extractor. The ConSERT¹¹1https://github.com/yym6472/ConSERT is fine-tuned with the dialogues data, to make the sentence representation more task-oriented and applicable to clustering. The output of [CLS] token is utilized as feature of each utterance for further clustering.

Clustering We use K-means (Krishna and Murty, 1999) to group utterances, and each cluster is treated as a dialogue action. For purity, annotator may check and slightly modify the clusters. Our practice shows the clusters are of high quality and require limited human efforts.

Dialogue standardization aims to standardize the dialogue by mapping each utterance to a specific dialogue action. Inspired by Yu et al. (2021), we exploit a retrieval-based method, which retrieves clustered utterances that are most similar to the given input utterance, and label the input based on corresponding clusters. Compared with traditional classification methods (Kowsari et al., 2019), such design has two advantages: 1) retrieval-based method is more suitable for our scenario where instance numbers of each cluster are imbalanced. 2) retrieval-based method can adapt to modification of actions (e.g. create new action or remove existing action) without having to retrain the model.

Specifically, as Figure 2 shows, given an input utterance $x$, we first use BM25 algorithm (Robertson and Zaragoza, 2009) to recall top $k$ utterances $\{x_{1},...,x_{k}\}$ from all clustered utterances. Further, we exploit a BERT-based text similarity computation model $S$ to rerank the utterances and select the utterance $\hat{x}$ with highest similarity to $x$, which can be denoted by:

where $S(x,x_{i})$ denotes the similarity between $x$ and $x_{i}$. The text similarity computation model is based on BERT, and takes concatenation of $x$ and $x_{i}$ as inputs (The input sentence is ([CLS], $x$, [SEP], $x_{i}$)). It first encodes the concatenated utterances into continuous representations, and computes similarity with output of the [CLS] token (denoted by $\boldsymbol{h}_{1}$ in the following) as follows:

where $\boldsymbol{W}$ and $b$ are trainable weight parameters.

TaskFlow is a tree with dialogue action as nodes, and the edges between nodes describe how the conversation proceeds (Refer to Figure 5 for a TaskFlow sample). We construct TaskFlow from the standardized dialogues, each of which can be treated as a sequence of user/staff actions. It’s intuitive to construct TaskFlow by directly inserting action sequence of each dialogue into a tree. However, due to the dynamic nature of dialogues, the action distribution across the corpus would be too scattered, given a certain turn. Similar conversation fragments may lie in different turns of dialogues. Thus, instead of the direct way, we first build a N-gram model of dialogue actions which captures the local conversation pattern more accurately. Further, we sample high-quality action sequences from N-gram model, and the sampled action sequences are merged together, forming the TaskFlow.

Building N-gram Model Given an action sequence $A=\{a_{1},a_{2},...,a_{n}\}$, the probability can be approximated as follows:

where $a_{i:j}=\{a_{i},a_{i+1},...,a_{j}\}$ denotes the subsequence with $i$ as starting index and $j$ as ending index. Following maximum likelihod estimation, we compute the N-gram as:

where $C(a_{i:j})$ denotes the count of $a_{i:j}$ in corpus.

Sampling Action Sequence We sample action sequences based on the N-gram model. Specifically, the sampling process starts with [SOS] and ends with [EOS]. We follow the beam search method (Freitag and Al-Onaizan, 2017) but with a different strategy. In each step, we extend every partial action sequence in the beam with its top K actions. Once the [EOS] symbol is appended to a partial action sequence, it is removed from the beam, and a complete action sequence is sampled.

Generating TaskFlow TaskFlow is generated by simply inserting the sampled action sequences into a tree individually (sequence by sequence). The transition probability is recorded as the condition on the edge between action nodes.

Post-Processing With the generated TaskFlow, operational staff can easily determine 1) which action nodes require API calls (e.g., staffs need to lock the bike remotely), 2) which conditions on the edges require modification (e.g., API response be specific value). For example, if a specific user action node has multiple children (i.e., staff action nodes) with different semantic information, an API call is probably required and each edge may correspond to different API response. We manually add API call nodes and modify the conditions on the edge, if necessary. In this way, TaskFlow consists of three types of nodes, named user action node, staff action node and api call node respectively.

We randomly sample 150 dialogues for each scenario. Annotators with domain knowledge are asked to grade each dialogue by “-1”, “0” or “1”, and the grading criteria can be summarized as follows: (i) Score “-1” denotes that Chatbot can not handle user requirements correctly. (ii) Score “0” denotes that Chatbot can handle user requirements correctly, but may generate influent or incomplete response. (iii) Score “1” denotes that Chatbot can handle user requirements correctly and perfectly complete the conversation.

The statistical results of human evaluation is shown in Table 1, from which we can observe that: (1) TaskFlow has satisfactory performance in terms of understanding and meeting user requirements, which is the core function of task-oriented chatbot. For exmaple, in scenario Forget_to_Lock_bike, TaskFlow can correctly handle user requirements of 96.31% dialogues. (2) Due to diversified user expressions in real-world scenarios, TaskFlow may generate incomplete and influent responses (e.g., 12.82% of dialogues in Forget_to_Lock_bike). For example, users may complain or curse about unexpected brake failures.

To quantitatively explore the low-cost characteristic of our method, we build a traditional task-oriented chatbot in Forget_to_Lock_bike (denoted by Traditional in the following) following Yao et al. (2013), and compare the required human efforts in Table 2. Though achieving comparable performance, our TaskFlow is low-cost and require much less human efforts (i.e., 9 v.s. 21 person-days).

Specifically, the building process can be divided into four steps, and person-days (abbr. as pds) required by each step is recorded in Table 2. Since traditional system involves multiple modules, it requires large-scale data annotation and more person-days for training and deploying online. With the presented automatic framework, TaskFlow-based system requires much less annotation and human efforts, thus effectively reducing developer burden.

With an explicit manner, TaskFlow-based system naturally has the advantage of controllability and Interpretability. Specifically, conversations exactly follow the paths in TaskFlow while traditional dialogue policy module may generate unexpected dialogue act. We present the TaskFlow in Forget_to_Lock_bike and a concrete conversation in Figure 4. If the condition on edge is met, the execution engine will move along the edge and respond according to the staff action node. For example, with API Check_Status returning True, the engine moves to staff action #2 node and respond with 3rd utterance. In this way, unexpected response will never be generated, and the path clearly shows how the conversation is completed.

The above advantages also lead to better flexibility. TaskFlow can be easily modified while it’s hard to manually intervene traditional dialogue system. When encountering changes of customer service policy, traditional dialogue systems may be unavailable, and we need to retrain the model with new labeled data. In contrast, our TaskFlow can be edited and adapt to changes immediately (without training).