Unsupervised Query Routing for Retrieval Augmented Generation

We first use BertScore Zhang et al. (2019) to evaluate the similarity between $r_{m}$ and its upper-bound $r^{\star}$:

The intuition is that the higher the value of $s_{m}^{\text{Bert}}$, the better the quality of $r_{m}$.

Besides Similarity, we also care about the coherence between the query and the single-sourced response, i.e., whether $r_{m}$ can effectively solve $q$. Therefore, we use a LLM, e.g., Qwen2-max, to determine which one better solves $q$ in a pair of single-sourced responses. Without loss of generality, we define the paired single-sourced responses as $r_{a}$ and $r_{b}$. Taken $q$, the multi-sourced documents $\textbf{DOC}^{\star}$, $r_{a}$, and $r_{b}$ as inputs, the LLM determines the one with better quality³³3The prompt is shown in Appendix A, Figure 6.:

Through $\frac{M*(M-1)}{2}$ comparisons, we can obtain the overall ranking relationship (in ascending order) among $\{r_{m}\}_{m=1}^{M}$⁴⁴4We discard the illegal ranking output by LLMs (like ”$A>B,B>C,C>A$”).. We define the coherence score $s_{m}^{\text{Coh}}$ as the index of $r_{m}$ in the overall ranking. The larger the index, the higher the score.

Having obtained $s_{m}^{\text{Bert}}$ and $s_{m}^{\text{Coh}}$ of $r_{m}$, we devise a normalization schema to get the overall score of $r_{m}$, which helps to mitigate the impact of varying dimensions of multiple metrics. For example, we calculate the mean $\mu^{\text{Bert}}$ and standard deviation $\sigma^{\text{Bert}}$ of the similarity score across all examples. Then, the normalized similarity score and coherence score are calculated as follows:

The overall score of $r_{m}$ is calculated as:

After obtaining $\{s_{m}\}_{m=1}^{M}$, we sort them in ascending order. We use the indices of $\{s_{m}\}_{m=1}^{M}$ in the ordered sequence as the label, denoted as $\mathbf{\pi}\in\mathcal{R}^{M}$. $\mathbf{\pi}$ actually represents a permutation of $\{1,2,\cdots,M\}$, indicating the priority of routing $q$ to $T_{1},\cdots,T_{M}$, respectively. This kind of label enables us to create a listwise ranking loss to effectively leverage the priority relationships between each pair of tools.

We use four public open-domain QA datasets, including NLPCC-MH Wang and Zhang (2018), CDQA Xu et al. (2024), WebQA Chen et al. (2019), SogouQA⁵⁵5https://aistudio.baidu.com/datasetdetail/17514, and a private QA dataset, named PrivateQA, for evaluation. The statistics of the dataset and the details of the construction of PrivateQA are shown in Appendix B.

We consider 3 search tools, i.e., Quark, Bing, and Google, for document retrieval. For each query, we retrieve $k=6$ documents. That is, in the upper-bound construction process, we only retain the top-ranked 2 documents for each search tool. We use Qwen2-max and GPT4 for response generation.

Each test example is in the form of (query, answer). For each query, we use the different combinations of LLMs and search tools to obtain the retrieval-augmented responses. We adopt the Correctness score implemented by Llama-index Liu (2022) as the metric to evaluate the quality of generated responses. For each query, we repeatedly generate three times and calculate the average correctness score. Since we have considered 3 search engines and 2 LLMs, we have 16 RAG strategies. The result is shown in Table 2. In the vast majority of strategies, using more search tools will lead to better results.

We utilize our in-house data to construct the training examples. After filtering out illegal samples (§ 4.3), we finally obtain around 110k training examples. It should be noted that there is a considerable distribution discrepancy between the training and test examples. As a result, the evaluation of our method can be regarded as in a non-i.i.d. setting.

We initialize our routing model as GTE-large Li et al. (2023). Since our method is model-agnostic, we have also tried other backbone models, as shown in § 5.4. We utilize the AdamW optimizer with an initial learning rate of 5e-5 and a batch size of 32. The learning rate is linearly decayed to zero with a 10% warmup ratio over 10 training epochs. Our model is evaluated through 5 randomized experiments, and we report the average results. All experiments are conducted using the PyTorch toolkit on an Ubuntu server equipped with four V100 (32GB) GPUs.

We alter the proportions of training data to assess the scalability of our method. For each proportion of training data, we present the average result of five random experiments. We consider the following baselines for comparison: (1) Random-1 which randomly selects a tool from Quark⁶⁶6https://www.quark.cn/, Bing, or Google for RAG; (2) Random-2 which randomly selects a combination from Quark+Bing, Bing+Google, or Quark+Google for RAG; (3) Best1Outof3 which is the most effective one among Quark, Bing, or Google; (4) Best2Outof3 which is the most effective one among Quark+Bing, Bing+Google, or Quark+Google.

It should be noted that Best1Outof3 and Best2Outof3 may correspond to different specific retrieval strategies on different datasets. Correspondingly, for each query, our method will predict the score of each tool. Therefore, we consider Top1-Predicted and Top2-Predicted, where Top1-Predicted indicates that we choose the tool with the highest predicted score, and Top2-Predicted indicates that we choose the two tools with the highest predicted scores. Ultimately, we compare Top1-Predicted with Best1Outof3, and compare Top2-Predicted with Best2Outof3.

The result is shown in Figure LABEL:fig:overall_scaling_effect. The random baselines show an unsatisfactory performance. In addition, we have the following observations.

• •

  As the volume of training data increased, our method has demonstrated consistent and obvious performance improvements. Since our method eliminates the necessity of manual annotation, the sustained enhancement in performance highlights the effectiveness of leveraging a larger amount of real user queries, ultimately, it is promising to obtain more refined and satisfied outcomes in diverse applications.

• •

  The effectiveness of our method is observed consistently across various LLMs and multiple datasets, underscoring the robustness and stability of our approach. The key reason is that a wide variety of real user queries are directly utilized for the learning of query routing, which can effectively guarantee data diversity.

• •

  On NLPCC-MH and CDQA datasets, our method outperforms the benchmarking baseline, demonstrating that it has learned the routing mechanism to some extent. This capability enables it to assign queries to the most suitable search engine, resulting in superior performance. While our method does not outperform the best baseline on some datasets, we attribute this limitation to the still insufficient volume of training data. Actually, query routing among general-purpose search engines is an extremely difficult task, since even humans struggle to determine which search engine should be selected to handle a query when they have no knowledge about black-box search engines. Fortunately, since we have observed the data scaling effect, we have reason to believe that if we continue to expand the data volume, our method will show even greater improvements and achieve optimal results.

Overall, by systematically varying the size of the training dataset, we observe compelling scalability for our method, which demonstrates its potential for utilizing more data for training.

Our method uses real user queries for training. As a comparison, we reproduce existing supervised methods Shnitzer et al. (2023); Mu et al. (2024) that utilize annotated (query, answer) data for model training. More specifically, we use the training splits of CDQA and WebQA which contain 10k and 20k examples, respectively, to construct training examples (§ 3.1). Then, we use the obtained examples to train the routing models and evaluate them on the 5 test sets. By altering the types of training data, we observe the scaling effect of the learned models on the 5 test sets.

The result is shown in Figure 3. We have the following observations.

• •

  In the independent and identically distributed (i.i.d.) settings, specifically in the "CDQA $\rightarrow$ CDQA" and "WebQA $\rightarrow$ WebQA" scenarios, supervised methods demonstrate a pronounced scaling trend. This indicates that augmenting the volume of training data correlates with sustained performance improvements, which aligns with human intuitive understanding.

• •

  Conversely, in the other non-i.i.d. scenarios of supervised methods, the addition of more training data fails to yield similar performance gains. This suggests that traditional supervised methods struggle to generalize effectively in non-i.i.d. environments. Such findings highlight a critical limitation in existing approaches. That is annotated data is difficult to obtain, which greatly limits the diversity of data and thus restricts their generalization ability.

• •

  Our method demonstrates a pronounced data scaling trend across all datasets, although the trend on NLPCC-MH is not as obvious as the trend on other test sets. This is primarily due to the fact that the data scale utilized by our approach significantly surpasses that of traditional supervised methods. Such an expansive data scale effectively enhances the diversity of the training data. This characteristic underscores the innovation of our approach, which is capable of automatically curating a wide array of diverse training data, eliminating the need for both time and cost consuming human annotation.

We develop the following two ablated variants to validate the effectiveness of different metrics in the label construction step: (1) "w/ Similarity" means we only use the similarity score for label construction; (2) "w/ Coherence" means we only use the coherence score for experiments. The result is shown in Table 3. Both "w/ Similarity" and "w/ Coherence" show a performance drop. In addition, we have the following observations.

• •

  The performance of "w/ Similarity" surpasses that of "w/ Coherence," indicating that the similarity metric is more effective. It is important to note that the similarity metric employs multi-sourced responses as the anchor, whose reliability has been validated by our experiments. This suggests that the similarity metric is a more objective measure, resulting in labels that are more robust. In contrast, the coherence metric relies solely on the knowledge within the LLM itself. However, due to inherent knowledge limitations in LLMs, it can generate incorrect information, leading to reduced effectiveness compared to the similarity metric. As evidence, we observe that LLMs may generate illegal ranking outputs, such as $A>B,B>C$, but $C>A$, in the coherence score calculation. These illegal outputs account for approximately 9% of the population.

• •

  Although not as effective as the Similarity metric, the Coherence metric contributes to our method. The best result is achieved when combining two metrics. One possible explanation is that Similarity and Coherence assess different aspects of the generated single-sourced responses. Specifically, similarity measures how closely a single-sourced response aligns with its upper-bound, while Coherence evaluates the rationality of a single-sourced response in relation to the input query. As a result, these two metrics can effectively complement one another.

We also investigate the impact of different backbone models. Besides GTE-large, we have also evaluated Roberta-large and Qwen2-0.5B, as they possess a comparable number of parameters to GTE-large. The result is shown in Table 4. We observe a minor result difference between GTE-large and Roberta-large. This is because they belong to the same type of models, and after training, they will all converge towards the distribution of the training data. However, Qwen2-0.5B shows a large performance drop compared to GTE-large and Roberta-large. We speculate that such decoder-only small models are less effective for regression tasks. Due to time and computational resource limitations, we are unable to try larger models, such as Qwen2-4 and Qwen2-7B, for experiments. Finally, we choose GTE-large as the backbone of our work.

We additionally assess the quality of the generated training datasets through manual review. We randomly sampled 200 automatically constructed examples, and manually evaluate the average accuracy of the labels generated by different metrics. The result is shown in Table 5.

The Similarity metric provides more accurate labels compared to the Coherence metric, as confirmed by our experiment (§ 5.4). By integrating both metrics, our weighted overall metric yields the highest accuracy in labeling. Given the automated nature of our method, we consider an 86% accuracy rate to be satisfactory.

We conduct a further analysis of the constructed training labels. According to the label $\pi$, we categorize the queries into three distinct groups: those should be assigned to Quark, Bing, and Google, respectively. Then, we tally the high-frequency words presented in the queries of each group and visualize high-frequency words in each group, as depicted in Figure 4. In the "Bing" group, terms such as "earthquake", "MacBook Pro", and "Hong Kong stock" demonstrate a high frequency, indicating that Bing has good support for news hotspots as well as significant political and economic events. In the "Google" group, high-frequency words include "python, open-source, 5G", etc. This indicates that Google is better able to handle programming, technology, and other queries that require strong professionalism. This is quite intuitive, as Google, renowned as the world’s leading search engine, offers the most extensive internal documentation available, enabling the platform to effectively tackle a diverse array of professional issues.