Exploiting LLMs’ Reasoning Capability to Infer Implicit Concepts in Legal Information Retrieval

Due to theirs generation capability and knowledge across various domains, LLMs are utilized to generate the key terms which represents the general topics of the query. Legal terms related to the query are extracted using zero–shot prompting techniques on LLMs. Since the experimented dataset is the COLIEE dataset with the original version written in Japanese, the prompting sentence will be written in Japanese to avoid information loss. Furthermore, to facilitate the processing of LLMs’ output, the prompting sentence includes additional instructions for LLMs output in JSON format. The prompting pattern and instruction in English version are described in listing 1, the one with Japanese version is presented at the listing 3 in the Appendix.

To simplify the experiments and verify the impact of the expanded information, the collection of legal terms will be directly concatenated with the query at the lexical layer. Since this concatenation disrupts the semantic integrity at the end of the query, lexical-based retrieval models will be most appropriate for this query–expansion type. The BM25 (?) model which is one of the most commonly lexical-based ranking models is utilized. The process of using BM25 to rank this type of expanded queries will be described in section 3.3.

Directly concatenating legal terms into the query does not ensure semantic coherence for the query, which may cause confusion for semantic–based ranking models. To preserve the semantic integrity, the legal–style oriented query reformulation is performed by zero–shot prompting technique on LMMs. The suggested prompting pattern written in English is detailed in listing 2, the Japanese one is showed at the listing 4.

The set of reformulated queries will be used to train the semantic matching model, further supporting information about the legal speciality of the content mentioned in the original query. The specific method to leverage the prompting results from LLMs will be described in section 3.3.

To combine the information from both prompting methods described in section 3.1 and 3.2, both lexical-based ranking and semantic-based ranking models will be utilized and ensembled.

Assuming the input query is denoted as $\mathbf{q}$ and the legal corpus, denoted as set $\mathbf{D}$, including $m$ legal documents: $\mathbf{D}=\{d_{1},d_{2},...,d_{m}\}$. After the legal-term extraction, the obtained terms form the set $T_{q}=\{t_{1},t_{2},...,t_{n}$}. These terms are concatenated with the query at the lexical level to create a new query:

The BM25 model is employed to calculate the lexical relevance between the expanded query $\mathbf{q}_{\text{term-expand}}$ and the legal document $d_{i}$. The relevance score calculated by the BM25 model is denoted as:

For the semantic-based ranking component, we utilized ranking models based on the BERT architecture, fine-tuned by sequence classification tasks. Experimenting with the simplest backbone model such as BERT will help assess the contribution of the additional information from the query expansion process. Two distinct BERT models are employed in this proposed retrieval system, one for ranking the original query and the other for the reformulated query. These two BERT models are described as follows:

• •

  Let $\mathbf{q}_{\text{origin}}$ be the original query. The first semantic ranking model denoted as $\text{BERT}_{\text{origin}}$, evaluates the semantic relevance between the original query $\mathbf{q}_{\text{origin}}$ and the legal document $d_{i}$, denoted as:

  $$\mathbf{R}_{ori}=\text{BERT}_{\text{origin}}(\mathbf{q}_{\text{origin}},d_{i})$$

• •

  Let $\mathbf{q}_{\text{reformulated}}$ be the reformulated query, derived from LLMs. The second BERT model, denoted as $\text{BERT}_{\text{reformulated}}$, evaluates the semantic relevance between the reformulated query $\mathbf{q}_{\text{reformulated}}$ and the legal document $d_{i}$, denoted as:

  $$\mathbf{R}_{\text{reform}}=\text{BERT}_{\text{reformulate}}(\mathbf{q}_{\text{reformulated}},d_{i})$$

The architecture of the retrieval system, which includes three ranking models, is illustrated in Figure 1. The final relevance scores from all three ranking models will be weighted ensembled using the equation 1.

where:

• •

  $\mathbf{R}_{\text{ori}}$ represents the correlation score between the original query and the article as calculated by the BERT-based ranking model.

• •

  $\mathbf{R}_{\text{bm25}}$ is the correlation score between the query, which has been concatenated with legal term outputs from LLMs, and the article, as calculated by the BM25 ranking algorithm.

• •

  $\mathbf{R}_{\text{reform}}$ is the correlation score between the query that has been re-described using LLMs and the article, also calculated by the BERT-based ranking model.

• •

  $\alpha$, $\beta$, and $\gamma$ are the weights assigned to the BERT model with the original query, the BM25 model with the legal–term expanded query, and the BERT model with the re-described query, respectively. The optimal values of these three weights are selected through a grid-search process on a validation set.

The legal document retrieval task is indeed about ranking the documents in the law corpus based on the relevancy between each document and the input query. Meanwhile, the training process for the BERT model uses the objective function of the Sequence Classification downstream task, classifying the (query-article) pair into positive or negative labels. Therefore, directly using the relevance score from the BERT model often does not yield the highest effectiveness for the retrieval system. To address this drawback, after obtaining the $R_{\textit{final}}$ for each (query-article) pair, a post-processing phase will be conducted. In the post-processing phase, the relevance scores for all (query-article) pairs corresponding to the same query will be min-max normalized. Subsequently, a common optimal threshold will be determined through a grid-search process on the validation set. All articles with a relevance score exceeding this threshold will be selected as relevant to the query. Specifically, the post-processing procedure for producing the final set of relevant documents is described in Figure 2.

To evaluate the effectiveness of the proposed retrieval system, the COLIEE 2022 ¹¹1https://sites.ualberta.ca/~rabelo/COLIEE2022/ and COLIEE 2023 ²²2https://sites.ualberta.ca/~rabelo/COLIEE2023/ datasets for the Statute Law Retrieval task will be utilized. Both datasets employ a corpus derived from the Japanese statute law, containing $782$ legal provisions.

The labeled set of COLIEE 2022 and COLIEE 2023 respectively includes $887$ and $996$ instances. The content of queries in both datasets often describes a legal statement, legal situation or a real–life legal scenario. Table 2 summarizes the number of samples and the length of each query through the statistical indices minimum, maximum, and average.

In this section, the implementation, training, inference, and ensemble of model results are discussed. Firstly, for the BM25 model, we utilize the rank-bm25 library³³3https://github.com/dorianbrown/rank_bm25. Before being processed by the BM25 model, articles and queries written in Japanese are word–based tokenized using the Konoha library ⁴⁴4https://github.com/himkt/konoha?tab=readme-ov-file and the MeCab tokenizer. The prompting process is performed on GPT–4 (?) and Gemini (?).

The training dataset provides sample which contains a query and a list of gold relevant articles. For training the BERT-based ranking model, the training samples are transformed into pairwise training samples. The transformed set contains a query and an article with two labels $0$ or $1$ representing not–relevant or relevant. The negative samples are chosen from top candidates according to the BM25’s relevance score. To achieve the optimal ratio between negative and positive samples, the top–30 most relevant candidates according to the BM25 model are utilized for creating negative samples for the queries. The Huggingface library ⁵⁵5https://huggingface.co is employed for creating and training the BERT-based model with the downstream task sequence classification. The pre–trained Multilingual BERT ⁶⁶6https://huggingface.co/google-bert/bert-base-multilingual-cased is being used for initializing the BERT-based retrieval model. The experimented models are fine–tuned with the training data in $3$ epochs. There are two ranking BERT-based models in the proposed retrieval system which are the ranking model fine–tuned with the origin query and the ranking model fine–tuned with the re–described query. The results in section 4.3 will demonstrate that using two separate BERT-based ranking models for original and re-described queries is more effective than a single fine-tuned BERT model handling both.

Before the training phase, a validation set is separated from $20\%$ of the labelled set, and the remaining $80\%$ of samples are formed into the training set. After the training the retrieval model, a grid–search process is conducted on the validation set for choosing the best $\alpha$, $\beta$, $\gamma$ and threshold.

The recall score of two term–expansion methods are presented in the table 4. According to that table, the recall scores when using the term–expanded queries instead of the original one are significantly higher at all top–k levels, with both Gemini and GPT–4 terms–expansion. This proves that adding legal terms will make the queries more lexically relevant to the gold standard articles, facilitating the BM25 model to rank more accurately. Additionally, the recall score of the Gemini–based term–expansion achieved a higher recall score than the GPT–4. Therefore, the BM25’s relevance score with Gemini’s expanded queries is used in the final retrieval system.

According to the implementation detail, the retrieval system is experimented with the COLIEE 2022 and 2023 datasets. Because the BERT’s training sample pair is generated by filtering from BM25’s candidates, the relevance score derived from BERT–based ranking model need to be ensembled with the relevance score from BM25 model to achieve full potential. The table 7 shows the results of three variants of retrieval system including:

• •

  (1) The system that ensemble BM25 model with term-expanded queries and the BERT model with original queries.

• •

  (2) The system that ensemble BM25 model with term-expanded queries and the BERT model with reformulated queries.

• •

  (3) The system that ensemble BM25 model with term-expanded queries, BERT model with original queries and another BERT model with reformulated queries.

The retrieval performance of the system (2) is lower than system (1). This decrease occurs because using LLMs to reformulated queries make the query’s distribution be changed significantly. However, the combination of three models achieved the highest F2–score which is $0.8449$ for the COLIEE 2022 dataset and $0.7643$ for the COLIEE 2023 dataset. This observation suggests that despite having different distribution with the original query, the reformulated query contributes different–perspective information for the retrieval system.

The table 5 (COLIEE 2022) and table 6 (COLIEE 2023) show the F2, precision and recall of the retrieval system (3) and results of all teams that participated the competition that years. As indicated on the table, the proposed retrieval system achieved an F2 score that was $2.49\%$ higher than the best-performing team in COLIEE 2022, and $0.3\%$ higher than CAPTAIN team (best performance team) in COLIEE 2023. This result demonstrates the effectiveness of retrieval system which utilizes legal-specific knowledge extracted from LLMs and the way for integrating the knowledge into the retrieval system.