Federated Retrieval Augmented Generation for Multi-Product Question Answering

Domain-specific QA has seen significant advancements across various fields, addressing the unique challenges posed by specialized knowledge and terminology. Research efforts have focused on developing tailored methods and datasets for domains such as biomedical Gu et al. (2021), physics Chen et al. (2023), finance Wu et al. (2023), and legal Cui et al. (2024). These works have contributed to improving QA accuracy and relevance within their respective fields. In the context of product-related QA, which is most relevant to our work, efforts have been more limited. Notable among these is the dataset in Liu et al. (2023) which focuses on Microsoft product queries. However, this dataset primarily consists of yes/no questions, with only a small portion requiring more complex generative text answers. Our work extends this line of research by addressing multi-product QA in enterprise software ecosystems. We focus on more complex, cross-domain queries that often require integrating knowledge from multiple products - a scenario common in enterprise settings but underexplored in current literature.

Retrieval augmented generation (RAG) has recently emerged as a powerful approach for enhancing the performance of LLMs in knowledge-base QA tasks. RAG combines the strengths of retrieval-based and generation-based methods to produce more accurate and faithful responses. Lewis et al. (2020) introduced the foundational RAG model, which retrieves relevant documents and conditions its output on both the retrieved information and the input query. Subsequent works have further improved it with Guu et al. (2020) developing REALM for joint training of retriever and generator, and Karpukhin et al. (2020) introducing dense passage retrieval for improved efficiency. Recent research has explored RAG in domain-specific contexts. Head et al. (2021) adapted RAG for scientific literature, while Khattab et al. (2022) investigated its application in customer support settings. Our work extends this line of research by introducing a novel multi-domain RAG framework that addresses the specific challenges of enterprise systems, where queries often span multiple products and require integration of diverse knowledge.

Multi-domain document retrieval presents unique challenges, particularly in enterprise product settings where information is often distributed across diverse and overlapping knowledge sources. Research in this area has focused on developing methods to accurately retrieve relevant information from multiple sources. Federated search approaches Shokouhi and Si (2011) enable querying multiple distributed indexes simultaneously, while domain adaptation techniques Shi et al. (2020) handle transfer across diverse search domains. Recent work leveraging LLMs for retrieval resource selection Wang et al. (2024b) has also shown strong zero-shot performance. Our work extends these efforts by introducing a stochastic gating mechanism combined with federated search, tailored for RAG-QA pipelines in complex environments with cross-product queries.

To effectively estimate the query-domain relevance scores. we leverage a query-domain router $\mathcal{F}:Q\rightarrow\left[0,1\right]^{m}$, mapping from the space of queries $Q$ to the $m$ product domains as a multi-label classification task. We use a Transformer model Vaswani (2017), specifically a variant of BERT, fine-tuned for our multi-domain classification task: $\mathcal{F}(q)=\sigma(W+\text{BERT}(q)+b)$, where $\text{BERT}(q)$ is the contextualized representations of query $q$ at [cls] token; $W$ and $b$ are learnable parameters; and $\sigma$ is the sigmoid activation function. As this is a multi-label classification task, we employ a binary cross-entropy loss for each domain, summed over all domains. At inference, we estimate the query-domain relevance likelihood with the trained domain router: $\mathcal{F}(q)=\left[p_{1},p_{2},\ldots,p_{m}\right]$.

To address the challenge of balancing exploitation of high-confidence domains with exploration of potentially relevant but less certain domains, we introduce a stochastic gating mechanism with adaptive threshold control. This approach allows for dynamic adjustment of the search space based on the Query-Domain Router’s confidence and the inherent uncertainty in multi-domain RAG-QA. We define an adaptive threshold $\tau(q)$ for query $q$ based on the entropy of the domain probability distribution $p$: $\tau(q)=\tau_{0}(1-\frac{-\sum^{m}_{j}{p_{j}\log{(p_{j})}}}{\log{(m)}})$, where $\tau_{0}$ is the base threshold hyperparameter; and $[p_{1},\ldots,p_{m}]$ is the vector of domain probabilities output by the router. We utilizie stochastic gating function $\mathcal{G}:M\times Q\rightarrow\{0,1\}$, with $M$ as the domain space and $Q$ as the query space, to facilitate domain selection and introduce exploration. This function is defined as $\mathcal{G}(q,j)=\text{Bernouli}\left(\min{\left(1,p_{j}/\tau(q)\right)}\right)$, where $\mathcal{G}(q,j)$ is the domain $j$-th selection for query $q$ based on the Bernouli sampling.

To facilitate efficient and effective retrieval of relevant documents across multiple product domains, we employ a bi-encoder architecture for our Query-Document Retriever. This model generates dense vector representations for both queries and documents, enabling rapid similarity computations in the embedding space. Our retriever embedding model $E$ is based on the Sentence-BERT Reimers (2019) with shared weights for query and document encodings, generating dense vector representations $E_{\theta}(q)$ and $E_{\theta}(d)$ for query $q$ and document $d$.

We fine-tune the retriever model on our multi-domain dataset using a contrastive learning approach with a symmetric supervised variant of the InfoNCE loss Oord et al. (2018), incorporating supervised relevance labels and symmetry, which we find particularly effective for query-document retrieval tasks in multi-domain settings. The retriever loss function is $\mathcal{L}_{r}=-(\mathcal{L}_{q2d}+\mathcal{L}_{d2q})/2$, where $\mathcal{L}_{q2d}$ and $\mathcal{L}_{d2q}$ represent the query-to-document and document-to-query directional losses. The $\mathcal{L}_{q2d}$ is computed as follows and the $\mathcal{L}_{d2q}$ can be obtained similarly.

For a given query $q$, we define the set of active domains $\mathcal{A}(q)=\{j\in M|\ \mathcal{G}(j,q)=1\}$. For each active domain $j\in\mathcal{A}(q)$, we retrieve the top-k documents $D_{j}=\{d^{1}_{j},\ldots,d^{k}_{j}\}$ and their respective query-document relevance scores $S_{j}=\{s^{1}_{j},\ldots,s^{k}_{j}\}$ obtained from the bi-encoder retriever detailed above: $s^{i}_{j}=s(q,d^{i}_{j})=E(q)\cdot E^{T}(d^{i}_{j})$. Next, we aggregate these to a unified domain-aware retrieval scoring $U(j,q,d^{i}_{j})=\mathcal{F}(q)[j]\cdot s(q,d^{i}_{j})=p_{j}\cdot s^{i}_{j}$, where $U(\cdot)$ is the unified multi-domain ranking score function for query $q$, domain $j$ and retrieved document $d^{i}_{j}$ at position $i$ in this domain. The final multi-domain ranked set of documents with federated search $D^{*}$ is obtained by selecting the top-k documents across all the active domains: $D^{*}=\arg\max_{k}{\{U(j,q,d^{i}_{j})|j\in\mathcal{A}(q),d^{i}_{j}\in D_{j}\}}$. These top-ranked documents from multiple domains are then augmented to the prompt and fed to LLM for the product QA.

The corpus is derived from the publicly available Adobe Experience League (ExL) documentation¹¹1https://experienceleague.adobe.com/en/docs, focusing on three key products: Experience Platform (AEP), Target, and Customer Journey Analytics (CJA). These web-pages provide comprehensive information on product concepts, capabilities, troubleshooting guides, and usage instructions.

The data preparation process involves several steps: ① Web Crawling: We employ a custom crawling script to extract content from the ExL web-pages. This script navigates through the documentation, capturing textual information while omitting images and converting clickable and in-section links to plain text for consistency. ② Initial Segmentation: The extracted content is initially segmented based on HTML header tags. This approach creates distinct sections that typically correspond to specific topics or tasks within each documentation. ③ Document Chunking: To optimize the corpus for efficient retrieval and context preservation, we implement the following chunking strategy. Each web-page is divided at every header level, creating initial chunks that align with the document’s logical structure. If any section exceeds a pre-defined token limit (512 tokens), we utilize LangChain’s hierarchical splitting approach based on a specified character list. This method prioritizes maintaining the integrity of paragraphs, sentences, and words, ensuring that semantically related content remains together as much as possible.

Our dataset comprises query-document pairs from three Adobe product domains (AEP, Target, and CJA). We employed two complementary approaches for data creation: ($i$) Subject Matter Expert (SME) Dataset: Product experts manually created query-document pairs based on respective ExL web-pages for each domain. They wrote queries for documents extracted from web-pages and annotated the relevance of each pair based on their product expertise. ($ii$) Synthetic LLM-Assisted (SLA) Dataset: To ensure comprehensive coverage, we leveraged GPT-4 to generate queries for chunked documents extracted from ExL web-pages. Product experts from each domain subsequently reviewed these query-document pairs to guarantee accuracy and relevance. For cross-domain data creation, we followed a similar process where product experts first identified ExL web-pages with overlapping documentation across products; GPT-4 then generated queries for these cross-domain web-pages; and product experts reviewed the queries for relevance to the cross-domain documentation content. This approach ensured that positive document pairs per query were designed to span different domains, enhancing the dataset’s utility for multi-domain product RAG-QA research.

To enhance the dataset with both positive and challenging negative examples, we employed a systematic approach for document pairing. For each question in the dataset, we utilized two strategies: (1) pairing the question with other document chunks from the same web-page as the golden document, and (2) when insufficient negative pairs were available from the original page, sampling document chunks from URLs closely related to the web-page containing the golden document. This method ensures a diverse and representative set of negative examples. Finally, we leveraged GPT-4 to annotate the relevance of each query-document pair. Using the prompt detailed in Appendix (Figure 6), GPT-4 assigns binary labels (Yes/No) to the relevance of all pairs.

Our dataset encompasses questions, documents, corresponding source web-page URLs and titles, and annotations across the Adobe AEP, Target, and CJA domains. The questions fall into two main categories: (1) "What-is" or "Where-is" questions about product concepts (e.g., "What is a union schema?", "What is an audience?"); and "How-to" questions about usage instructions (e.g., steps for "adding services to a datastream" or "looking up a sandbox"). Table 1 provides key data statistics for uni-domain and cross-domain datasets, including the count of question-document pairs, average lengths of questions and documents, and the ratio of positive pairs in the datasets.

To comprehensively assess the performance of our multi-domain RAG-QA pipeline, we employ a variety of evaluation metrics targeting both retrieval accuracy and response quality: ($i$) Retrieval Accuracy: For evaluating retrieval performance across multiple domains, we use the Acc@Top1 metric. This metric represents the percentage of queries for which the golden (most relevant) multi-domain documents are correctly retrieved as the top-ranked candidate. Our focus on the first document is motivated by recent RAG studies Liu et al. (2024); Xu et al. (2024) showing that the top-ranked document, when added to the prompt, most significantly influences the LLM’s response. ($ii$) Response Quality: To assess the quality of generated answers for product QA, we employ Relevancy and Faithfulness analysis. In the former, we incorporate GPT-4, following the prompting strategy in Zheng et al. (2024), to evaluate the relevancy and helpfulness of the generated responses to queries. Given the domain-specific nature of product QA, we also utilize the RAGAS²²2https://docs.ragas.io framework Es et al. (2023) to assess the faithfulness of generated responses. In this metric, we decompose each response into individual statements and cross-check them against the ground truth documentation for each query with the help of GPT-4. The faithfulness score is computed as the percentage of statements that GPT-4 recognizes can be directly inferred from the provided context. Detailed prompts for these metrics are available in the Appendix.