Wizard of Search Engine

The goal of this task is to identify the general intent of each user utterance. There have been several similar tasks (Qu et al., 2019). However, the proposed user intent taxonomies are either too general or not well mapped to user behavior in TSEs. Considering previous work (Radlinski and Craswell, 2017; Azzopardi et al., 2018) while making a connection to TSEs, we define a new user intent taxonomy, as shown in Table 2. Most intents can map to the corresponding user behavior or operation in TSEs, except for “chitchat” and “request-rephrase”, because TSEs do not have conversations and the outputs are always organized as SERPs. The algorithmic problem of ID in CIS is defined as learning a mapping: $\{C^{:\tau},X^{\tau}\}\rightarrow i$. That is, given the context $C^{:\tau}$ and the current user utterance $X^{\tau}$, the goal is to predict the user intent $i$ of for the utterance $X^{\tau}$.

Conversations usually contain more content than necessary in order to ensure the integrity of grammar and semantics, etc. Ellipsis and anaphora are common in conversations, so the current user utterance itself is usually not semantically complete. This is especially true as conversations go on and when the users are not familiar with the topic, or unclear about the information goals themselves. It is challenging to fetch documents that are relevant to the intent in the current user utterance by simply issuing the current user utterance to the TSE. To this end, similar to (Voskarides et al., 2020), we define KE as a task to extract keyphrases from the current user utterance as well as the previous conversational context, which can help TSEs to retrieve relevant documents for the current user utterance. Formally, KE is defined as learning a mapping: $\{C^{:\tau},X^{\tau}\}\rightarrow K^{\tau}$.

To connect users to the right information with maximal utility, a good CIS system should be able to take appropriate actions at the right time, e.g., helping the users to clarify their intents whenever unclear, and providing information in the right form by tracking user expectations. AP has been well studied in task-oriented dialogue systems in a closed-domain setting (a.k.a. dialogue policy) (Peng et al., 2017). However, AP is not yet well addressed in CIS. Although there have been some investigations about what system actions are necessary in CIS, to the best of our knowledge, it has not yet been put into practice (Radlinski and Craswell, 2017; Azzopardi et al., 2018). By recalling the operations in TSEs, we introduce a system action taxonomy in Table 3. The AP task is formulated as learning a mapping: $\{C^{:\tau},X^{\tau},Q^{\tau},D^{\tau}\}\rightarrow a$. Note that this task also takes the candidate queries $Q^{\tau}$ and candidate passages $D^{\tau}$ fetched from TSEs as input, as they will influence the action the CIS system should take. For example, to decide whether “no-answer” is the action to be taken, the CIS system must check whether the answer is available in $Q^{\tau}$ and/or $D^{\tau}$.

In TSEs, a list of suggested or related queries is usually presented to users to help them clarify intents or find similar intents. In CIS, we cannot always simply list the suggested or related queries to users, as this is not natural in conversations. Previous studies cast this problem as a problem of asking clarifying questions, and model it as either selecting a question from a question pool prepared in advance (Aliannejadi et al., 2019) or generating a clarifying question for a given keyword query (Zamani et al., 2020). However, neither option fits in our formulation. Besides, how to recommend similar search intents rather than asking clarifying questions is neglected. We introduce the QS task as a step in the CIS pipeline. Depending on the predicted actions from the upstream AP task, QS aims to select queries fetched from TSEs that can be used by the downstream task to generate clarifying questions or recommend similar search intents. Formally, QS is defined as learning a mapping: $\{C^{:\tau},X^{\tau},Q^{\tau},(i,a)\}\rightarrow Q^{\tau}_{s}$, where $Q^{\tau}$ is set of suggested or related queries fetched from TSEs with the keyphrases from KE as queries, and $i$ and $a$ are the user intent and system action from ID and AP, respectively.

Passage or document ranking has long been studied in TSEs (Wu et al., 2019b). Recently, this has also been investigated in a conversational context (Dalton et al., 2020b). The situation is a bit different for PS in CIS where the context is not only about the proactive questions from users but also the clarifying questions or recommendations from the systems. And the goal is not always to find the relevant passages for the user questions but also non-relevant passages that contain related topics in order to do clarification or recommendation sometimes (White and Roth, 2009). Given the same input as in QS, the PS task is to select passages based on which to generate system responses by the downstream task: $\{C^{:\tau},X^{\tau},D^{\tau},(i,a)\}\rightarrow D^{\tau}_{s}$, where $D^{\tau}$ is a set of passages fetched from a TSE with the same keyphrases from KE as queries.

RG is a fundamental component for many NLP tasks, e.g., chitchat (Ren et al., 2020a), TDS (Pei et al., 2020), conversational question answering (Baheti et al., 2020). The goals or formulations of RG vary across tasks. The role of RG in our formulation of CIS is to translate system actions and predictions of the above tasks into natural language responses. For example, if the system action is “clarify”, the RG part needs to generate a clarifying question by referring to the selected queries. In this work, we only focus on free text responses. The formulation of RG is defined as a learning problem: $\{C^{:\tau},X^{\tau},Q^{\tau},D^{\tau},(i,a,Q^{\tau}_{s},D^{\tau}_{s})\}\rightarrow Y^{\tau}$.

First, we collect a set of 1,196 search intents from a search log of a commercial search engine. Each search intent is based on a specific search session. For a given search session, e.g., “[anecdotes, anecdotes in the world, Tolstoy, Tolstoy Wikipedia, Tolstoy movie]”, the workers are asked to infer the search intent behind it using their imagination, and write down a description in one of the following forms: (1) specific intent, e.g., “You are interested in anecdotes about famous writers in the world like Tolstoy. Try to learn about his/her basic information, life story and representative works, and so on.” (2) exploratory intent, e.g., “You want to find some material about anecdotes. Describe the features of anecdotes you are interested in. Try to learn about the introduction of the anecdotes and ask for links.” (3) multiple intents, e.g., “(1) or (2) + You also want to find a movie that is based on his/her works. Ask for information about the director, plot, links and so on.”

The conversation flows as follows. (1) The searcher randomly picks a search intent and starts by asking questions directly or by uttering a greeting. When sending a message, the searcher is required to select a label from Table 2 that can best describe the intent of the message. (2) Whenever the wizard receives the message from the searcher, s/he needs to extract keyphrases from the conversation history that are used to fetch results from a search engine. Then s/he has to select a label from Table 3 that reflects the action s/he is going to take. After that, s/he needs to select the relevant queries and/or documents, based on which to formulate the responses. (3) The conversation repeats until the searcher ends the chat (after a minimum of 7 turns each). For each turn, both the searcher and wizard can send multiple messages at once by clicking the “send another message” option. After collecting data in this way, the goal is to then replace the wizard with a learned CIS system that will speak to a human searcher instead.

It took 24 workers 3 months to build the dataset. The final dataset we collected consists of 1,905 conversations with 37,956 utterances spread over 1,196 search intents. In total, the dataset contains 12 different Intents and 23 different Actions, covering a variety of conversation topics such as literature, entertainment, art, etc. Each conversation contains 7 to 42 utterances. The average number of utterances per conversation is 19.9, and the average number of turns per conversation is 9.2. Each utterance has an average of 27.3 words. We divide the data into 705 conversations for training, 200 conversations for validation, and 1,000 for testing. The test set is split into two subsets, test(seen) and test(unseen). The test(seen) dataset contains 442 overlapping search intents with the training set with new conversations. Test(unseen) consists of 500 search intents never seen before in train or validation. The overall data statistics can be found in Table 4.

As shown in Figure 3, we propose an architecture that relies mainly on transformer encoders and decoders.

We use binary cross-entropy loss for KE, QS and PS and cross-entropy loss for ID, AP and RG. To solve complex tasks with a deep model, a large volume of data is needed. We get around this problem by first pre-training the model on four datasets from related tasks, as shown in Table 5.

WebQA is a dataset for single-turn question answering. For each sample, it provides a simple question in natural language, a set of candidate passages, and an entity answer. DuReader is for single-turn machine reading comprehension. The questions are mostly complex and the answers are summaries of the relevant passages. It also has question and answer types, which can be used for the pre-training of T1 and T3. DuConv and KdConv are for knowledge grounded conversations. DuConv has structured knowledge in triples while KdConv has free text sentences. Since the sentences in KdConv are usually short, we use it to pre-train T4. DuConv also has annotations of topic words in conversations which can be used to pre-train T2. For KdConv, we pre-train T2 by predicting the overlapping tokens in conversational contexts and answers. We do not pre-train T2 using WebQA or DuReader, because they are single-turn and the questions are very short usually. After pre-training, we fine-tune the model on the WISE data.

We report the results of all methods on WISE; see Table 6. We consider three settings: (1) With pretraining (i.e., WISE in Table 6). (2) Without pretraining (i.e., WISE-pretrain in Table 6). (3) With pretraining and using the ground truth of ID, KE, AP, QS and PS (i.e., WISE+GT in Table 6). We also report the performance of WISE on the test (unseen) and test (seen) datasets, and on different actions. The results are shown in Table 7 and 8.

From Table 6, there are two main observations. First, WISE+GT achieves the best performance in terms of all key metrics. As we can see in Table 6, WISE+GT achieves 20.4% and 24.2% in terms of BLEU and ROUGE for RG, outperforming the strongest model WISE by 5.8% and 7.3%, respectively. This shows that the human labels of ID, KE, AP, QS, PS do benefit RG.

Second, WISE achieves a higher performance than WISE-pretrain in terms of most metrics, indicating that the pretrain phrase is of great benefit for the performance of WISE. For example, the BLEU and ROUGE score of WISE for RG are 13.6% and 15.9%, much higher than those of WISE-pretrain, respectively. The reason is that the model can leverage external knowledge in the pretrain phrase, which boosts the final performance.

From Table 8, we obtain three main findings. First, for the action ‘Clarify’, the best performance in terms of KE, AP and QS is achieved w.r.t. ‘Clarify choice’ and in terms of RG w.r.t. ‘Clarify yes-no’. The worst performance in terms of KE and QS is achieved w.r.t. ‘Clarify yes-no’ and in terms of AP and RG w.r.t. ‘Clarify open’. For response generation w.r.t. ‘Clarify yes-no’, WISE can almost copy the answer from a single query, which is easier than response generation w.r.t. ‘Clarify choice’ and ‘Clarify open’, which different queries, like ‘city’ from ‘Beijing’ and ‘Jinan’.

Second, for the action ‘Answer’, the best performance in terms of KE and AP is achieved w.r.t. ‘Answer-open’ and in terms of PS and RG w.r.t. ‘Answer-fact’. The worst performance in terms of all tasks is achieved w.r.t. ‘Answer-opinion’. For response generation w.r.t. ‘Answer fact’, WISE can copy rather than summarize from multiple passages w.r.t. ‘Answer open’ and ‘Answer opinion’.

Third, the performance of PS has a high correlation with the final RG task w.r.t. ‘Answer’, while the performance of QS does not seem to be correlated w.r.t. ‘Clarify’. As we can see in Table 8, an F1 score of 23.3% in PS and BLEU score of 12.6% in terms of RG is achieved w.r.t. ‘Answer opinion’, and a higher F1 score (30.0%) in terms of PS and a higher BLEU score (16.4%) in terms of RG w.r.t. ‘Answer fact’. For response generation w.r.t. ‘Answer’, large portions of the output can be copied from passages, and thus the performance PS is crucial to the final RG performance. The F1 score of WISE is 10.9% in QS w.r.t. ‘Clarify yes-no’, which is lower than 29.6% w.r.t. ‘Clarify choice’; the performance in RG is better, e.g., 28% vs. 20%. It is more likely that WISE has to copy from multiple queries for ‘Clarify choice’ and from a single query for ‘Clarify yes-no’, leading to performance differences for QS and RG, w.r.t. ‘Clarify’.

We split the test dataset based on the occurrence of conversational background in the training dataset: (test (unseen) means no occurrence, and test (seen) means occurrence); the results are shown in Table 7. Generally, the performances of WISE on all tasks is best on the test (seen) dataset, and worst on the test (unseen) dataset. For example, it achieves 29.6% in terms of F1 for PS in test (seen) dataset, which is higher than 24.9% in test dataset and 20.5% in test (unseen) dataset. WISE has been trained on the same conversational background in the test (seen) condition, and has never seen the conversational background in the test (unseen) condition.

We analyze the effects of different tasks in WISE, w.r.t. ID, KE, AP, QS and PS, and the results are shown in Table 9. We remove each task, resulting in five settings; ‘-xx’ means WISE without module ‘xx’. Generally, removing any module will result in a drop in performance for RG on all metrics, indicating that all parts are helpful to WISE. Without QS, RG drops sharply on all metrics. Specially, it drops 4.2% in terms of BLEU for RG, which means QS is essential to RG. While without PS, RG drops a little compared to QS. Specially, it drops only 0.9% in terms of BLEU for RG. The performance of PS is far from perfect, only 24.9% in F1 score; omitting it has limited impact. In addition, removing ID, KE or AP harms the performance of QS and PS, indicating that all previous modules for QS and PS help for the performance of QS and PS. Specifically, performance drops by 10% and 5.7% in terms of F1 score for QS and PS when removing KE: if the model does not extract the accurate keyphrase, it can not select the most relevant queries and passages.

To analyze the effects of different pretrain datasets, we conduct an ablation study (Table 10). WISE uses four external datasets for pretraining; each time we remove one, we use the others for pretraining. Hence, we have four conditions. In Table 10, ‘-xx’ means pretraining WISE using datasets other than ‘xx’.

The results show that all datasets are helpful for pretraining; removing any dataset results in a performance drop for RG. The results drop in terms of all metrics when pretraining without the DuReader dataset. The results drop by 5.0% and 5.9% in terms of BLEU and ROUGE for RG. The drop after removing pretraining with DuReader is far larger than after removing any of the other individual pretraining dataset, indicating the importance of DuReader for pretraining, as shown in Table 5. Interestingly, sometimes removing a dataset will improve the performance of ID and AP. For example, -duconv achieves 35.8% in terms of F1 in ID, higher than the 34.1% for WISE. ID and AP predict labels among small sets, which are relatively easy compared to the other tasks.