Optimized Quran Passage Retrieval Using an Expanded QA Dataset and Fine-Tuned Language Models

In context of expanding the number of Muslim populations worldwide (2.04 billion in 2024), there Is a growing demand for understanding the holy Quran through a reliable question-answering (QA) system capable of providing exact explanations and answers from the Qur’an [9]. In recent years, the Qur’an QA 2023 shared task dataset has highlighted the complexity of this task, as previous methods have shown limited accuracy in retrieving relevant Qur’anic verses [19]. Many traditional QA models face difficulties with the linguistic nuances of Classical Arabic and the specificity required in Holy Qur’an [10]. the study of this work builds upon this challenge by expanding the existing dataset and employing advanced language models to improve retrieval accuracy [9]. The contributions in this paper are as follows:

• •

  Dataset Manipulation and Expansion: We expanded the original QA dataset by generating new questions through rephrasing and categorization, resulting in a significantly larger and more diverse set of 1895 questions.

• •

  Accurate Language Model Fine-Tuning: We fine-tuned multiple transformer models on the expanded dataset, achieving notable improvements in passage retrieval accuracy, particularly with the AraBERT-large model.

The layout of this article is outlined in the following manner: Section 1 explore into the reasons and importance of improving Qur’anic question-answering systems. Section 2 gives a summary of previous research, outlining major progress and obstacles in the area. In Section 3, The research methodology is described, including data collection, expanding the dataset, cleaning the data, and finetuning the language model. The setup utilized to assess model performance is explained in Section 4’s experimental design. Section 5 highlights and discusses the outcomes of experiments are presented, showcasing the efficiency of the method. In conclusion, Section 6 of the paper presents suggestions for upcoming studies.

The field of question answering (QA) for Holy Qur’an, specifically Qur’anic passage retrieval, has attracted a lot of attention from researchers because of the distinct linguistic and contextual difficulties presented by the Qur’an [21]. The task requires accurately retrieving relevant verses to answer both factoid and non-factoid questions, often requiring systems to bridge the linguistic gap be- tween Modern Standard Arabic (MSA) and Classical Arabic [20]. Additionally, systems must be able of recognizing questions that have no answers within the Qur’anic text, thereby requiring strong mechanisms for zero-answer scenarios [25]. The development of effective QA systems for Holy Qur’an remains a challenging endeavor, as the richness of the Arabic language and the need for contextual understanding demand advanced modeling techniques [14].

Transformer-based language models, such as AraBERT, CAMeLBERT, and AraELECTRA, have shown promise in handling Arabic language tasks [2]. However, existing models often struggle with limitations stemming from insufficient and imbalanced training datasets, which impact the models’ ability to generalize effectively to unseen queries [18, 12, 13]. The Qur’an QA 2023 shared Task, for example, highlighted the necessity of utilizing external resources and data augmentation strategies to improve performance. Sarhan and Elkomy [9] addressed these challenges by leveraging ensemble learning techniques, combining dual-encoder and cross-encoder architectures, and employing transfer learning on external datasets such as TyDI-QA and tafseer. Their ensemble strategy improved prediction stability and effectiveness, achieving a MAP score of 25.05% for passage retrieval. Despite these advancements, their study underscored the persistent need for dataset expansion and more robust fine-tuning approaches to tackle the linguistic complexities of Qur’anic QA tasks [9].

The work of Alawwad et al [1] focused on enhancing Qur’anic passage retrieval through the use of pre-trained models fine-tuned on specialized datasets like Tafseer and TyDI-QA [6]. They merged thresholding mechanisms to manage unanswerable questions and demonstrated the benefits of ensemble learning for improving performance in low-resource settings. Although their approach showed considerable success, achieving improved performance metrics, it also pointed to the critical role of dataset quality and quantity in achieving consistent model accuracy. The limitations of existing datasets and the models’ dependency on external resources highlight the need for further dataset expansion and optimization strategies. Mahmoudi et al [19] proposed a multi-task transfer learning approach that employs models like AraElectra and AraBERT, utilizing both unsupervised and supervised fine-tuning to adapt to the Qur’anic context. They implemented tech-niques like TSDAE and SimCSE to produce top-notch sentence embeddings, greatly improving the models’ ability in reading comprehension and passage retrieval. Their study showed that custom sentence embeddings and thorough model fine-tuning can result in actual enhancements. Nonetheless, the study also emphasized that further advancements require larger and more diverse datasets to fully Understand the complexities of the language and context of the Quran. This work builds on these prior efforts by addressing the pressing need for dataset expansion and more effective model fine-tuning [27]. A significant increase was achieved the size of the Qur’an QA dataset from 251 to 1895 questions, employing strategic question rephrasing to improve data diversity and model robustness [26]. The proposed approach involves extensive experimentation with multiple transformer- based models, including AraBERT, CAMeLBERT, and AraElectra, fine-tuning them on this enriched dataset. By doing so, the aim of this work is to enhance the models’ ability to handle complex queries, manage zero-answer cases, and improve overall passage retrieval accuracy, contributing to the advancement of QA systems for Holy Qur’an [1].

This section details the structured process followed to create and improve the dataset utilized in Qur’anic Question Answering (QA) systems. The methodology includes key phases: data collection, dataset expansion, data cleaning, and model fine-tuning. Each phase is thoroughly designed to ensure the quality, reliability, and efficacy of the dataset, establishing a strong foundation for robust QA performance (see Figure 1).

This paper experimental design involves fine-tuning several pre-trained Arabic Language Models to tackle the tasks of Question Answering (QA) and Passage Retrieval (PR) in the context of Qur’anic texts. These models, including AraBERT, CAMeLBERT, BERT, Roberta, and AraELECTRA, have been adapted to better address the unique linguistic and semantic challenges presented by the Qur’an. The fine-tuning process, utilizes expanded dataset of 1895 questions derived from an initial set of 251 questions.

Transfer learning was employed to enhance model performance, leveraging the pre-trained knowledge of each model and adapting it through a targeted finetuning process. Moreover, implement ensemble learning techniques, combining predictions from multiple fine-tuned models. This approach improves overall answer accuracy and robustness by leveraging the strengths of each model while mitigating individual weaknesses. Furthermore, a thresholding mechanism was used to filter out low-confidence predictions, effectively managing zero-answer cases by discarding uncertain or ambiguous results. The evaluation framework uses the following metrics to measure model performance:

• •

  Mean Average Precision (MAP): Measures the precision of each rank position and compares it to the average in all queries, allowing a complete ranking quality assessment.

• •

  Mean Reciprocal Rank (MRR): Assesses the position of the first useful responses in the result list and indicates how well the model finds the correct answer.

• •

  Recall:Measures such as recall@5 and recall@10 are used to evaluate the effectiveness of the model in capturing relevant information in the highest rankings.

The evaluation results indicate that fine-tuning various pre-trained models significantly enhanced the performance metrics for question answering and passage retrieval tasks. This section discusses the results presented in Tables 1 ,2 and 3.

Table 1 compares the Mean Average Precision (MAP@10) and Mean Reciprocal Rank (MRR) measurements of each model before and after adjustment. AraBERT-base the most notable improvement was MAP@10, which increased from 0.22 to 0.36, and MRR, from 0.37 to 0.59. These findings indicate that the retrieval accuracy and ranking accuracy have improved significantly, and show that the model has improved its ability to effectively capture relevant passages. The Roberta model, moderate improvements (MAP@10 was 0.05 to 0.12, MRR was 0.01 to 0.17), but it benefited from fine tuning. AraBERT-large remained relatively stable, with only slight increases in MAP@10 and MRR, suggesting that the model was already well-tuned for these tasks or required additional modifications for further improvements. CAMeLBERT-base and AraELECTRA- base demonstrated balanced enhancements, with moderate improvements in both metrics. BERT-squad-accelerate, on the other hand, achieved significant progress, with MAP@10 increasing from 0.07 to 0.25 and MRR from 0.12 to 0.40, highlight the importance of dataset expansion and fine-tuning approach.

Table 2 illustrates the recall performance of various models at different cut-off points. AraBERT-base demonstrated a notable improvement in recall metrics, with Recall@5 increasing from 0.25 to 0.37 and recall@100 rising from 0.30 to 0.50. This indicates a stronger capacity for the model to retrieve relevant passages in the top ranks. In contrast, Roberta showed limited enhancements, with Recall@5 improving only from 0.10 to 0.18, suggesting potential challenges in grasping the nuances of Qur’anic text. The AraBERT-large model exhibited moderate improvements across recall metrics, with Recall@5 moving from 0.31 to 0.34 and maintaining consistent performance in other recall metrics. CAMeLBERT-base achieved strong results, improving Recall@5 from 0.32 to 0.36 and showing potential in handling more complex queries, while AraELECTRA- base displayed significant gains, especially in Recall@15, with an increase from 0.42 to 0.57. BERT-squad-accelerate maintained stable recall values at lower cut-off points but significantly performed better in handling unanswerable questions, as showed by a No Answer Recall of 0.75, up from 0.25. This highlights its effectiveness in addressing queries without clear answers. Table 3 presents precision metrics across various cut-off points. AraBERT- base showed significant improvements, with Precision@5 rising from 0.18 to 0.25 and Precision@100 increasing from 0.01 to 0.06. These enhancements underscore the model’s effectiveness in accurately identifying relevant passages. Roberta showed only small improvement in precision, with Precision@5 moving from 0.05 to 0.10, which corresponds with its overall average performance. Both AraBERT- large and CAMeLBERT-base demonstrated improved precision, particularly at higher cut-off points, indicating their ability to deliver more relevant results, with CAMeLBERT-base achieving Precision@5 of 0.27 compared to its base score of 0.23. The AraELECTRA-base model displayed balanced precision gains, showing a notable increase from 0.21 to 0.30 at Precision@5.

Meanwhile, BERT-squad- accelerate excelled in handling no-answer scenarios, attaining a No Answer Precision of 0.75, significantly higher than its base score of 0.25, highlighting its strength in addressing unanswerable questions.

This study presents an effective approach to improving Qur’anic passage retrieval in question-answering systems by fine-tuning pre-trained Arabic Language Models (LMs) on an expanded dataset of 1,895 questions. Models like AraBERT, CAMeLBERT,Roberta , AraELECTRA, and BERT ,optimized using transfer learning, showed improved ability to realize the difficulties of both the Qur’anic text and user queries. Ensemble learning further boosted accuracy and developments, while a thresholding mechanism ensured reliable answers and managed zero-answer cases. The paper results highlight the significance of thorough datasets and sophisticated model structures in creating strong QA systems for Holy Qur’an, contributing both to Natural Language Processing research and providing a valuable resource for Muslims. Future work is recommended to explore additional data sources and further architectural refinements to address challenges like semantic understanding and unanswerable queries (zero answers).

Heartfelt gratitude is extended to AiTech AU, AiTech for Artificial Intelligence and Software Development (https://aitech.net.au), for funding this research, providing technical support, and enabling its successful completion.