DomainRAG: A Chinese Benchmark for Evaluating Domain-specific Retrieval-Augmented Generation

To alleviate the hallucination problem of language models, Retrieval-augmented generation (RAG) strategy Gao et al. (2023); Zhu et al. (2023) is proposed by providing external references to LMs to help them provide more accurate and factual answers. In particular, RAG approaches usually devise a retrieval model to collect relevant documents or passages according to user queries from the corpus. Then, these retrieved references are fed together with the user queries into the downstream language models to generate answers. Traditional approaches Guu et al. (2020); Lewis et al. (2020) in this area mainly focus on supporting language models with limited parameters such as BERT, BART, and T5. Recently, with the development of Large language models, more and more researchers have paid attention to improving the RAG performance based on large language models by considering the retrieval frequency Lazaridou et al. (2022); Ram et al. (2023); Jiang et al. (2023); Liu et al. (2023), designing delicate CoTs Yu et al. (2023); Trivedi et al. (2023), training the retrieval models and language models together Asai et al. (2023); Lin et al. (2023), comprising the retrieved references to fit the input length limit Arefeen et al. (2023); Xu et al. (2023); Li et al. (2024). Research Ren et al. (2023) has shown that by incorporating the retrieved documents, RAG models do respond with few mistakes.

Previous studies in the retrieval-augmented generation area mainly conduct experiments on open-domain QA datasets such NQ Kwiatkowski et al. (2019) and HotpotQAYang et al. (2018) using Wikipedia as the retrieval corpus. Though this general evaluation setting can somehow reflect the quality of answers generated by the RAG models, it fails to analyze the abilities of these models from different perspectives such as intent understanding and faithfulness to the references. Besides, since Wikipedia is widely used in the pre-training of language models and the information in the retrieved documents may have been learned by LLMs, it is questionable whether RAG models really utilize the references to answer questions instead of their intrinsic knowledge. Recently, Chen et al. Chen et al. (2024) alleviated this problem by proposing a specialized RAG benchmark to analyze the four disentangled fundamental abilities of different large language models. However, it still focuses on the open-domain QA, without considering the LLM’s ability under in-domain situations. To thoroughly assess the abilities of RAG models, in this paper, we propose to leverage an in-domain document corpus collected from the enrollment websites of a Chinese university to evaluate the capabilities of RAG from multi-aspects.

To acquire the document corpus for this scenario, we crawled web pages from the admission official website with official permission. We not only extracted their text contents but also reserved the original HTML contents to facilitate the construction of the structural QA dataset. Given the lengthy nature of web pages, we further split the text contents of each web page into passages using a sliding window of 256 length and 128 overlap. The numbers of web pages and passages are 1,686 and 14,406 respectively. Finally, we created two document corpora, a text corpus and an HTML corpus. The evaluation datasets are built by initially being generated from powerful generative models (ChatGPT or GPT-4), then being corrected manually.

$\bullet$ Extractive QA Dataset. We first randomly sampled document passages from the text corpus. These passages were then incorporated into the prompt designed for ChatGPT, which generated question-answer (QA) pairs based on the provided passages. To ensure the in-domain nature of questions, we specifically instructed ChatGPT to generate questions that cannot be answered without providing external information. The selected passages can be directly considered as positive references for the corresponding questions in the dataset.

$\bullet$ Conversational QA Dataset. To build question-answering conversations, we began by choosing documents with substantial content, primarily focusing on the introduction web pages of each school. We then utilized ChatGPT to generate domain-specific questions according to each passage within the selected documents. This process resulted in a collection of question-answer pairs associated with each document. To test the conversational intent understanding ability of RAG models, we simplified each question in the QA pair list (except the first one) by removing entities that duplicate the preceding questions. The revised QA list can be regarded as a vanilla conversation. Furthermore, we derived multiple conversation samples from a given vanilla one. Specifically, at the $t-$th step, the QAs from the previous $t-1$ steps were considered as historical conversations, and the current $t$-step QA was treated as the question and the golden answer.

$\bullet$ Structural QA Dataset. To assess the understanding capabilities of RAG models on structural information (we focus on table structures in this paper), we first selected web pages containing table structures. Then, we offered the HTML contents of these web pages to ChatGPT and instructed it to generate QA pairs where the answers are derived from the table information. To accommodate the input length limits of ChatGPT, we preprocessed the HTML content by removing irrelevant elements, such as HTML comments, script tags, etc. For each QA pair, we provided both the HTML and corresponding pure text content of the positive document. This approach not only allows us to evaluate the models’ ability to analyze and comprehend HTML structures but also to compare the effectiveness of using HTML structural information versus pure texts for solving problems.

$\bullet$ Faithful QA Dataset. To test the faithfulness of LLMs to domain-specific knowledge, we first provided document contents to GPT4 and prompt it to generate QA pairs that rely solely on external expert knowledge, rather than common-sense information. This step is similar to the process used in creating the extractive QA dataset. Furthermore, to ensure the generation quality, we manually filtered out the QA pairs that could be answered using knowledge contained within LLMs themselves. Finally, we modify the answer-related information in the positive references to build anti-references and corresponding anti-answers.

$\bullet$ Noisy QA Dataset. To evaluate LLMs’ robustness to noisy information in provided references, we expanded upon the extractive QA dataset to create the noisy dataset. Concretely, for each piece of data, we randomly sampled several irrelevant passages from the text corpus to construct noisy information. During the test experiments, we varied the number of irrelevant passages selected and combined them with positive references to build external references with different noise ratios.

$\bullet$ Time-sensitive QA Dataset. Since the dataset is static, it is difficult to evaluate the abilities of RAG in answering real-time questions. Inspired by Dhingra et al. (2022), we focused on generating questions that have different answers at different timestamps. To indicate the timestamp of each question, we included a “date” attribute in each data sample. It is challenging for ChatGPT to automatically generate time-sensitive question-answer pairs that require rich prior domain knowledge. Therefore, we manually design possible questions and identify answerable document passages to build answers and positive references.

$\bullet$ Multi-doc QA Dataset. To address complex questions that can not be fully answered by simply extracting information at the entity level, it becomes necessary to aggregate information from multiple relevant documents. To build a dataset that evaluates such ability, we follow a specific approach. First, we identified a set of relevant documents that share similar topics or themes, such as introductions to relevant institutes or majors. These documents serve as the basis for generating the dataset. Next, we provide the text contents of these relevant documents to GPT4, which generates questions that require answers derived from multiple document contents.

The statistical information of our datasets is demonstrated in Table 1. The noisy dataset is derived from the extractive dataset, its average lengths of queries and answers are the same as the extractive. There are two items of average answer length (Avg. A Len) of the faithful dataset where the first is the golden answer, the second is the anti-answer.²²2The anonymous link of our dataset is provided here: https://github.com/ShootingWong/DomainRAG.

We first conducted experiments using the following external knowledge settings,

(1) Close-book: No external domain-specific knowledge was provided to assess whether LLMs could solve these expert problems themselves.

(2) Golden reference: We provided human-annotated positive references for LLMs to explore the upper bounds of their abilities.

(3) Retrieved reference: Simulating real-world applications of RAG models, we provided them with retrieved documents. We chose BM25 Robertson and Zaragoza (2009) and BGE-base-zh-v1.5 Xiao et al. (2023) as two classical retrievers to represent sparse and dense retrieval.

(4) Noisy reference. To test the robustness of LLMs on noisy external knowledge, we provided different levels of irrelevant references blended with golden references. We also investigated the impact of the position of golden references within the noisy references on RAG models’ performances.

(5) Structural reference. In the experiments on the structural QA dataset, we provided two versions of golden references, i.e., HTML and pure texts, for LLMs to evaluate the abilities to analyze HTML structures and compare the effect of structural information versus pure texts for RAG tasks.

6) Anti-reference. In the faithful QA dataset, we provided both golden and anti-references for LLMs in the same question to compare the faithfulness of LLMs in utilizing expert external knowledge.

We selected six commonly used LLMs, including Llama2-7B-chat, Llama2-13B-chat, Llama2-70B-chat, Baichuan2-7B-chat, Baichuan2-33B-32k, ChatGLM2-6B-32k, and gpt-3.5-turbo-1106 to compare their abilities comprehensively. Note that we chose the Baichuan2-33B-32k version with a built-in general retrieval system to further assess the effectiveness of general knowledge sources in our domain-specific scenario.

To evaluate model performance, we chose four widely used metrics: Two versions of exact-match, where the one assesses whether the ground truth answers are contained by predictions (EM), the one assesses whether the predictions are strictly the same as the answers (EMS); F1 is used to evaluate models in the perspective of term-matching; Rouge-L and GPT-4 evaluation (GE) are used to assess the performance of long-form answers, i.e., conversational and multi-doc datasets. For the GE metric, we prompt GPT to score whether the prediction is consistent with the answer from the three perspectives: factual consistency, redundancy, and deficiency. The predicted score should range from 0 to 5 and we normalized it to $[0,1]$.

The overall experimental results of the first three settings are shown in Table 2, and we analyze the following conclusions.

(1) In domain-specific scenarios, the knowledge contained within LLMs themselves may hard to tackle the user’s expert problems. The experimental results in the "Close Book" block confirm the poor performance of LLMs when faced with in-domain questions that go beyond their internal knowledge Additionally, the retrieval settings in the last four blocks demonstrate that external expert knowledge can provide more reliable information for LLMs in expert scenarios. Even when equipped with a built-in retrieval system like Baichuan2-33B-32k, the close-book results are significantly inferior to those obtained from retrieval settings. This finding reinforces the importance of domain-specific corpora over general knowledge sources.

(2) The BM25 retriever shows better generalization than the dense retriever. Interestingly, when using BM25, the results are generally better than the dense retriever. One possible explanation is that our application scenarios are long-tail, meaning they contain specialized knowledge that may not have been adequately covered during the pre-training of the dense retriever. However, BM25 is a spare retrieve with strong generalization ability. Therefore, the BM25 is also a good choice for RAG models in domain-specific application scenarios, especially if applications are more cost-sensitive.

(3) Dealing with long-form QA problems remains challenging for RAG models and warrants further investigation in the future. We notice that for conversational and multi-doc datasets, the improvement of retrieval-augmented results is not as significant as in other datasets. Especially for the multi-doc dataset, its results of providing golden references are still limited. These phenomena imply that accurately understanding the user’s current intents in conversational scenarios requires RAG models to possess strong abilities in analyzing complex relationships among historical information, the current query, and external references. Moreover, analyzing the interactions between multiple external documents is also a critical ability for RAG models and needs to be investigated further.

To evaluate the effectiveness of structural information for RAG models and analyze their abilities to comprehend knowledge in HTML format, we conducted the corresponding experiments on our structural QA dataset. It is worth noting that the whole HTML content of a web page is redundant and may contain some useless information about the web layout. Therefore, we proactively filtered out the information irrelevant to the valuable content of web pages. Nevertheless, the processed contents still exceed the maximum length of some LLMs, e.g., Llama. For simplicity, we directly truncated the provided information for LLMs that cannot handle lengthy texts. We expect that there are more elaborate techniques to tackle this problem. We provided two versions of web page content: one is pure text and the other one is HTML content for comparing the performance of RAG models on the different formats of external knowledge. The experimental results are presented in Figure 2.

Obviously, for Llama models and Baichuan2-7B, the predicted results from pure texts are more accurate than ones from HTML content. However, for Baichuan2-33B-32k, ChatGLM2-6B-32k, and GPT-3.5-turbo-1106, providing HTML content leads to better performance than providing pure texts. According to this phenomenon, we can draw the following conclusions: (1) Structural data contains valuable information beyond pure texts. Strong LLMs, such as GPT-3.5-turbo-1106, demonstrate better results when using HTML-format external knowledge, suggesting that the structural information of web pages complements textual content and helps LLMs understand web content and address user queries effectively. (2) Baichuan2-33B-32k, ChatGLM2-6B-32k, and GPT-3.5-turbo-1106 have stronger abilities in understanding and analyzing HTML contents than Llama-family models and Baichuan2-7B. The reason may be that these models have been pre-trained on data in the HTML format, which enables them to better comprehend the corresponding information. In the future, with more diverse formats of external knowledge, such ability is more and more important for LLMs to provide better experiences for users.

To assess the robustness of LLMs on noised references, we mixed the positive references with different amounts of noisy references, including 4, 9, 14, 19, and 24. Additionally, the position of the positive reference was varied, i.e.the first, the middle, and the last positions, to assess the impact of the reference order on RAG models. The experiments were performed on LLMs capable of processing long texts, i.e.Baichuan2-33B and GPT-3.5-turbo-1106, considering the potential issue of overlength when dealing with extensive external knowledge.

The results in Figure 4 indicate that both different positions of golden references and amounts of noise have a significant influence on the performance of RAG models. There are some interesting findings: (1) Lost in the middle is a common phenomenon. Placing positive references in the middle position of noisy references often leads to a significant decline in model performance. This phenomenon has also been indicated in recent studies Liu et al. (2024b), highlighting the importance of not only the quality of the provided knowledge but also its order. (2) More noise often leads to worse results. It is evident that the lines with high $NC$-values are generally below the line with low $NC$-values (“NC” is short for “Noise Count”). It is reasonable since excessive irrelevant knowledge may disturb the LLMs’ cognition, thus negatively affecting the generated results. Therefore, a high-quality IR model is also critical for RAG tasks. (3) Noise is not always bad. The results of “No Noise” are not always the best compared to those obtained from noisy references. The reason may be that compared to the “No Noise” setting where only one document is provided, the noisy references contain $NC+1$ external documents. The increased amount of provided knowledge may emphasize the confidence of LLMs in external knowledge, making them more inclined to rely on it when solving problems. To verify this assumption, we conducted an experiment, where the golden references were repeated to match the number of noisy references. This experiment partially supports this assumption as the repeated references outperformed all other settings in most situations. This observation provides some insights for future studies of RAG that repeating provided references may be beneficial for motivating LLMs to provide better results.

To assess the faithfulness of LLMs in external knowledge in out-of-domain applications, we provided the anti-references for LLMs to test whether they could generate anti-answers for these expert questions according to the external information. We compare the results with two different settings, in one we provided golden references and tested the performance of generating golden answers, other one is the close book setting. The comparison results are demonstrated in Figure 3.

We found that in the close book setting, LLMs significantly underperform the settings with external knowledge, further confirming the importance of external knowledge for this scenario. Additionally, whether or not external knowledge is provided, LLMs often tend to generate golden answers instead of anti-answers. This suggests that (1) LLMs still contain a certain of prior knowledge about the in-domain information. (2) There is some room for LLMs to improve their faithfulness in the right external knowledge. It is reasonable that LLMs could answer some questions according to their own knowledge, hence may impact their confidence in external information. Nevertheless, their own knowledge is static and may be out-of-date while knowledge in some in-domain scenarios, such as enrollment plans, will quickly change over time. These situations put forward a strong requirement for LLMs to distinguish and trust external knowledge correctly.