Learning to Relate to Previous Turns in Conversational Search

Our goal is to generate gold Pseudo Relevance Labels (PRL) for each historical query/term with respect to a given query. We assume that a relevant historical turn/term should make a positive impact when the current query is expanded with it. The pseudo labeling is based on the impact on retrieval results of a candidate turn/term when the latter is used to expand the query.

The turn-level generation procedure is introduced in Algorithm 1. We first use the current raw query to retrieve the ranked list of documents and calculate the base retrieval score (MRR in this case). Then we iteratively concatenate the query with each of the historical queries as a new reformulated query and get the corresponding retrieval score. If the second score is higher than the initial one, then the relevance of the historical query is labeled “positive”, otherwise, “negative”. Query expansion is produced by concatenating the expansion query $h_{i}$ with the current one $q_{n}$, i.e. $q^{exp}=q_{n}\circ h_{i}$, where $\circ$ means concatenation. In the same way, we can also create term-level pseudo labels for all the terms in the historical queries.

We assume the gold PRL as the results from an oracle classifier that can correctly separate the useful and harmful historical expansion turns/terms. Before proposing an approach to select useful expansion turns/terms, we first examine if the gold PRL can indeed improve retrieval performance by using the relevant expansion turns/terms. We formulate these query expansion forms as:

Fig. 2 shows the retrieval effectiveness scores with these types of queries on the TopiOCQA dataset. We can see significant improvement in retrieval effectiveness by expanding the queries with generated PRL with two retrievers: BM25 (sparse) (Robertson et al., 2009), ANCE (dense) (Xiong et al., 2020). In general, we see that expanding queries by all historical queries is beneficial, leading to improved effectiveness compared to the raw queries. This result is consistent with the observations in previous studies. However, when we use only relevant queries annotated by PRL, the effectiveness is much higher. This demonstrates clearly the potential benefit of selecting the relevant previous historical context and confirms our earlier assumption. We also find the expansion by the relevant tokens is more suitable with the sparse retriever while the turn-level expansion performs better with the dense retriever. The comparison between the two retrievers clearly shows the advantage of the dense retriever ANCE. Therefore, in our experiments, we will use ANCE as the retrieval backbone and relevant historical queries as expansion units.

Given the high impact of relevant queries for expansion, our problem is to develop an effective selection model to identify them. The principle we use is the selected expansion query can bring improvement in retrieval and be judged as relevant for the current query turn by a neural selection model.

So far, the main approaches to QR (Yu et al., 2020; Anantha et al., 2021; Wu et al., 2021; Voskarides et al., 2020) rely on external supervision signals, i.e. human-rewritten queries, to train a rewriting model. In practice, it is difficult to obtain such manually reformulated queries. Other methods expand the query with all historical queries (Qu et al., 2020; Yu et al., 2021; Lin et al., 2021; Kim and Kim, 2022), which will inject noise during expansion, as we showed in the previous section. To overcome these issues, our method learns a query selection model (selector) based on the pseudo-labeled data to select useful historical queries for expansion and produces an expanded query to be submitted to a retriever.

However, we also notice that there is a hidden dependency between the retrieval model and the expansion process. An off-the-shelf retriever such as ANCE has been fine-tuned with unexpanded queries. Fine-tuning it further for expanded queries can yield better results. Similarly, the selector also relies on a retriever to determine useful expansion queries. To account for the dependency, we design a multi-task learning framework to jointly fine-tune the selector-retriever (S-R) models through conversational search data.

The overview of our methods is in Fig. 3, which contains three main components: PRLs generation, selector model training with off-the-shelf retrieval, and multi-task S-R model jointly fine-tuning.

We want to use conversational search data to fine-tune the retriever. We expect that the fine-tuned retriever would perform better. However, our preliminary experiment indicates that directly using the selected expansion queries as the input for the fine-tuning would degrade the final performance of the fine-tuned retriever (Sec. 7.2). Thus, a different strategy used for leveraging the automatically labeled queries should be designed.

From the retrieval perspective, a historical query might be always potentially relevant to the current turn to some extent, depending on the fine-tuning stage of the retriever. In the previous strategy, we only keep the queries with positive labels and remove the others. This ”hard” selection strategy is static and does not allow for changes during fine-tuning. Instead, we will use a ”soft” weighting of historical queries, so that during fine-tuning, the status of a historical query may change depending on the fine-tuned retriever. In other words, we want to make the relevance labeling dependent on the retriever used.

To implement this idea, we propose a multi-task learning method to jointly fine-tune the selector and retriever using conversational search data. We take the query $q^{\text{all}}$ concatenating with all the historical queries as input and train a way to assign weights to each candidate query, while also trying to optimize the retrieval effectiveness. This corresponds to a ”soft” selection of candidate expansion queries embedded in the retrieval process. The expected advantage is that the retriever is optimized together with the soft selection of expansion queries, thereby alleviating the problem of data scarcity and yielding effective fine-tuning.

The joint training objective consists of two parts. One is the contrastive learning ranking loss for the retriever as shown in Eq. 2. Another one is the loss of selector training with the weight assignment strategy. The intuition is that if the retriever has the ability to distinguish relevant queries, it would be able to assign reasonable weights to the candidate expansion queries in $q^{\text{all}}$. The whole learning object of S-R is shown as Eq. 3. Since retrieval is the main task, $\alpha$ aims to moderate the effect given to selector training.

Datasets: We use four conversational search datasets in our experiments. The TopiOCQA (Adlakha et al., 2022) and QReCC (Anantha et al., 2021) with relatively large amounts of training samples are used for evaluating our supervised methods (i.e. with selector training), while the other two small datasets CAsT 2019 (Dalton et al., 2020) and CAsT 2020 (Dalton et al., 2021) are used for evaluating zero-shot retrieval (i.e. no further training). More details about the datasets are provided in Appendix A.1.

Since the experimental settings of existing works are not exactly the same as ours, it is unfair to directly compare them. Thus, we implement several strong baselines following the existing methods with the conditions described in Sec. 3. The comparisons for supervised methods include: (1) No expand: The query of the current turn without expansions. (2) QuReTeC (Voskarides et al., 2020): A token-level expansion method to train a sequence tagger to decide whether each term contained in the historical context should be added into the current query. (3) All expand (Yu et al., 2021): It expands the current query with all historical turns and input to the ANCE (Xiong et al., 2020) dense retriever. This is a widely used approach and the strongest baseline in the literature (Qu et al., 2020; Mao et al., 2022a; Kim and Kim, 2022). (4) Rel. expand (Human): It uses only the manually determined relevant historical queries to expand the current query. This method requires human annotations about topic (Adlakha et al., 2022), where we assume two turns are related if their ground-truth topics are the same. (5) Query Rewriter (Yu et al., 2020): This model is based on a GPT-2 (Radford et al., 2019) rewriter model fine-tuned on QReCC.

For zero-shot setting, in addition to the QuReTeC and Query Rewriter, the other compared methods include: (6) ZeCo (Krasakis et al., 2022): A token-level representation based method using Col-BERT (Khattab and Zaharia, 2020) dense retriever. (7) ANCE-ConvDR (Yu et al., 2021): The zero-shot version of ConvDR based on ANCE (Xiong et al., 2020) dense retriever. (8) ConvDR Transfer (Yu et al., 2021): Fine-tune ConvDR on conversational search data only using the ranking loss (Eq. 2). In particular, for a fair comparison, we implement all the compared systems by training them on TopiOCQA without external datasets to make them comparable to our methods, except for the results directly quoted from original papers.

We implement our selector and retriever models based on the BERT (Devlin et al., 2019) and ANCE (Xiong et al., 2020) using the PyTorch and Huggingface libraries. The retriever model follows the bi-encoder architecture (Karpukhin et al., 2020) and we fix the passage encoder during fine-tuning. More details are provided in Appendix A.2 and our released code¹¹1https://github.com/fengranMark/ConvRelExpand.

We first evaluate the effectiveness of conversational dense retrieval with our query selection methods for query expansion. Table 1 shows the overall results using queries reformulated in different ways and submitted to ANCE. We show three different versions of our method: “BERT PRL” selects useful historical queries using BERT, “ANCE PRL” selects the queries using ANCE, and “+ Weight Assignment” uses soft selection (i.e. by assigning a weight to each historical query according to their category during training). We have the following observations:

(1) Our proposed methods with useful expansions outperform all baseline methods on both datasets. Specifically, our best relevant expansions (Rel. expand) improve 19.1% and 20.6% on MRR and NDCG@3 over the main competitor that expands with all historical context (All expand) on TopiOCQA. The above result confirms that only part of the historical queries are useful for expanding the current query, while the remaining ones are noise that would hurt retrieval performance. Such observation is consistent with previous studies. It also outperforms the Query Rewriter method, validating our assumption that we can reformulate the query for conversational search relying on useful historical queries automatically determined rather than manually rewritten queries. In addition, our method is much more efficient: the inference speed of our selector is hundreds of times higher than Query Rewriter. Besides, the method of relevant expansions by human annotations (Rel. expand Human) is also better than expanding all queries (All expand), though it is clearly lower than the oracle method and our best method with weight assignment. This confirms our assumption that human-annotated relevant queries are not the best ones for expanding the current query, and a model can be trained to determine better useful historical queries than manual annotators.

(2) The results on QReCC exhibit a similar trend to those on TopiOCQA, but the difference between the baseline and upper bound (Oracle) is smaller. Thus its performance boost is not as strong as on TopiOCQA. This is mainly because the topic-switch phenomenon is uncommon in QReCC: the conversations contain fewer turns (12.3 in TopiOCQA v.s. 3.6 in QReCC), and the queries in a conversation mostly revolve around the same topic (Wu et al., 2021). However, in a more realistic setting with a longer conversation, we expect that topic-switching in conversation would appear more often, thus the utility of our approach would be more obvious.

(3) The oracle relevant expansions by gold PRL is still far ahead of that with our selector. To approach the upper bound, the PRL produced by the ANCE selector with specific strategies can further improve the retrieval scores. Thus, training a better selector is important in future work.

(4) To evaluate the performance of the selector, we report the precision, recall, macro F1, and accuracy of different selector training strategies on our generated PRL data in Table 2. We can observe that there is no big difference in both the retrieval and classification performance between using BERT PRL and ANCE PRL. By exploiting the weight assignment (W.A.) strategy, all the metric scores except the recall of the selector drop a lot. Since it obtains better search results, we conjecture that the recall score of the selector is more related to the retrieval results than precision.

In this section, we test the idea of joint fine-tuning of the selector and the retriever. The All expand and our Multi-task S-R method use $q^{\text{all}}$ as input for retriever fine-tuning, while the No expand and the remaining methods use $q^{\text{raw}}$ and $q^{\text{PRL}}$, respectively. Then the fine-tuned retriever might be more suitable for conversational scenarios. The overall results are shown in Table 3. Together with the observations in former Sec. 7.1, we have the following observations:

(1) Our multi-task S-R improves 5.5% and 9.5% over the strongest baseline (All expand) w.r.t MRR and NDCG@3, and approaches the performance of Rel. expand with gold PRL. This demonstrates the effectiveness of selector training as an auxiliary task. Different from the explicit selecting methods that boost MRR and NDCG@3, the implicit ones such as the All expand method and our multi-task S-R are good for Recall@100. This might be due to the fact that we use MRR as the retrieval score in Algorithm 1, so the Rel. expand with gold PRL will optimize MRR but not necessarily Recall@100.

(2) We find that the reformulated queries $q^{\text{PRL}}$ with only useful expansions cannot improve the performance of the dense retrievers after fine-tuning and even hurt it. From Table 3, we can see the Rel. expand method with human judgment and gold PRL are worse than the All expand method on recall. The predicted PRL obtained by ANCE is even worse, indicating that the selected queries are incompatible with fine-tuning. We also find that only All expand with $q^{\text{all}}$ and our Multi-task S-R can improve recall along with the fine-tuning procedure, while the Rel. expand method with $q^{\text{PRL}}$ keeps dropping from the beginning. Such phenomenon contradicts our intuitions and previous work (Mao et al., 2022a). A possible explanation is that the gold standard PRL is produced by the base retriever before fine-tuning, while the old gold standard PRL may no longer be the gold standard for the fine-tuned retriever. On the other hand, our multi-task learning method can mitigate this issue to some extent. Although the gold standard PRL remains the same, the selector is updated together with the retriever and it prevents the model from performance degradation. A potentially better strategy is to update the gold PRL based on the fine-tuned retriever in every epoch so that the query relevance is estimated in tight connection with the retriever. We leave this strategy to future work.

We also evaluate our approaches in the zero-shot scenario on two CAsT datasets and compare them with other dense retrieval methods. The overall results are shown in Table 4.

Our methods can be divided into two parts. The selector approach is directly applying the ANCE selector obtained from Sec. 7.1 based on TopiOCQA, to explicitly select useful expansion turns, then feed the reformulated queries into the ANCE retriever. Another approach is based on fine-tuned S-R retriever obtained from Sec. 7.2 via multi-task learning. We can see that our methods produce higher or equivalent performance than the compared methods on various evaluation metrics. This demonstrates their high applicability to a new dataset without further fine-tuning. From the oracle results for reference, we can see the superior quality of both the relevant expansion by humans and our gold PRL, which indicates the high potential of the upper bound of our produced PRL data. Besides, we find the scores of human annotation are still lower than our gold PRL. This confirms again our assumption that human-annotated relevant queries are not necessarily the best ones for expansions.

In this section, we make further analysis of some specific aspects of our methods.

We finally show one successful and one failed case to help understand more intuitively the behaviors of different expansion methods. We compare the results of different expansion judgments and also show the comparison of the retrieval effectiveness between expanding all historical queries and expanding only the relevant ones.

The corresponding results are shown in Table 7. For the successful case, the model is expected to understand the “name of the series of The Man Trap”. The first round query supplies crucial information “The Man Trap”, while the second asking for characters of the fiction would inject noise because the response for it involves names which is also the goal for the current query. Though the human annotator thinks both the first and second round queries are related to the same topic as the current one, the second round cannot provide useful information for retrieval. For the failed case, the model needs to retrieve “the members of the cast for the mentioned film”. Both the human judgment and the Gold PRL indicate some relevant queries in the context, and expanding them could bring improvements in retrieval effectiveness. However, the selector did not select any expansion queries, which should be avoided and improved. The potential reason is that it is not enough to leverage only the semantics between the current query and the relevant history queries for accurate selection. Besides, the retrieval effectiveness using Gold PRL as expansion is still better than expanding all contexts. Therefore, a better selector should be able to achieve higher effectiveness for this query.