Improving Vietnamese Legal Document Retrieval using Synthetic Data

In this subsection, we outline our comprehensive workflow for generating synthetic query datasets and fine-tuning retrieval models using Vietnamese legal texts. This workflow is visually represented in Figure 1, which illustrates the sequential stages from data collection to model fine-tuning.

Our methodology is divided into several key stages:

Stage 1: Collect legal text data. This initial stage involves the collection and preprocessing of legal documents. The documents are split into smaller passages suitable for further processing.

Stage 2: Generate queries using Llama 3 prompt with legal passages. Using the Meta Llama 3 model, we generate synthetic queries based on aspects identified in the legal text passages. This involves generating aspects from the legal texts and then crafting queries for these aspects.

Stage 3: Filter low-quality queries. We remove low-quality queries that explicitly refer to the input passage and those that are only shallowly relevant to the input passage. For queries that are only shallowly relevant, we use the BGE-M3 dense retriever to filter out synthetic queries that cannot recover their input passage within the top 40 retrieved results.

Stage 4: Query-as-Context Pre-training. We employ the generated queries to further pre-train our language model, focusing on enhancing its ability to understand and retrieve relevant passages.

Stage 5: Hard negatives mining for fine-tuning datasets. We mine hard negative examples to create a robust fine-tuning dataset, which is crucial for improving the retrieval model’s accuracy.

Stage 6: Fine-tuning. The final stage involves fine-tuning the pre-trained model with the generated data, optimizing it for the specific task of legal text retrieval.

Wang et al. [27] use GPT-3.5/4 to generate queries along with corresponding positive and hard negative passages by maintaining output diversity through two stages of generation: first generating retrieval tasks, then using them to generate query-passage pairs. As our objective focuses on creating a domain-specific dataset for Vietnamese legal text retrieval, relying solely on prompt engineering for this task would be complex and inefficient. Therefore, we opted to use collected legal text passages as input for an LLM to generate queries related to the content of each passage.

Our main source for collecting legal documents was thuvienphapluat.vn. The website hosts a wide range of legal documents, including Laws, Decrees, Circulars, Joint Circulars, Resolutions, Ordinances, Decisions, and the Constitution. After scraping both the metadata and full-text content, we preserved the hierarchical structure of each document, which typically includes chapters, sections, articles, and clauses. This structure was essential for maintaining context and ensuring that the text remained coherent after being split into smaller passages. Each passage retained crucial information, such as the document’s domain, title, header, and main content, which provided the necessary context for accurate query generation. This process resulted in 143,261 passages, which were used as input for the LLM to generate high-quality, contextually relevant queries.

For generating synthetic queries, we chose the open-source LLM Llama 3 70B [2] due to its strong performance, particularly in Vietnamese. Llama 3, trained on over 15 trillion tokens, outperforms many LLMs with a similar parameter count and demonstrates robust multilingual capabilities, making it well-suited for our needs.

To maintain diversity and relevance in query generation, we experimented with different prompting techniques. A direct approach, where the model generated questions without identifying distinct aspects of the passage, often led to less diverse and sometimes irrelevant queries. Through various prompt designs, we discovered that instructing the model to identify 1 - 5 different aspects covered in the passage and then generate a question for each aspect yielded the most relevant and diverse queries. In Figure 2, we show the prompt template used to generate synthetic queries from legal text passages. Examples of a generated query and its corresponding passage are shown in Table 1. A quantitative analysis of the generation method will be later discussed in Section 5.1.

Applying this method, we generated over 620,000 legal queries from 140,292 passages extracted from our curated Vietnamese legal text collection. We then employed the BGE-M3 dense retriever [4], which demonstrated strong zero-shot performance in our testing, to filter out queries whose corresponding passages did not appear in the top 40 relevant results. Additionally, we excluded queries that directly referred to the passage using terms like ”quy định này” or ”thông tư này”. This process results in a final dataset of 507,152 Vietnamese legal queries, covering a wide range of legal domains, which is then used to pre-train and fine-tune our retrievers.

We generated these synthetic queries using Llama 3 70B through Together AI’s free credits program, with the entire generation process costing approximately 200 USD.

Query-as-Context pre-training [26] is based on the observation that text spans within the same document can vary significantly in semantics, potentially weakening the effectiveness of traditional pre-training techniques. However, one limitation noted in this approach is that the T5 model used often produced a substantial number of unrelated query-passage pairs, which could diminish its overall effectiveness. This aligns well with the first part of our work — generating a legal query dataset — where we address this issue by utilizing an advanced LLM to produce queries and implementing a filtering process to remove irrelevant pairs, as detailed in the previous subsection.

Using the pairs of collected passages $x_{i}$ and corresponding legal queries $y_{i}$ generated by Llama 3, we apply the loss function from the Contextualized Masked Autoencoder (CoT-MAE) framework [29]. Specifically, the encoder reconstructs the passage $x_{i}$ using its unmasked tokens, while the decoder reconstructs the query $y_{i}$ by leveraging both its unmasked tokens and the contextual passage $x_{i}$. The training objective combines the encoder’s masked language modeling (MLM) loss and the decoder’s context-supervised MLM loss, ensuring that the model learns to effectively integrate the query and passage context during pre-training.

To verify the effectiveness of our pre-training, we fine-tuned a bi-encoder and ColBERT model on downstream retrieval tasks. Our fine-tuning process is based on a single-stage pipeline with hard negative mining, utilizing a comprehensive set of datasets.

For both models, we utilized the MS-MARCO passage ranking dataset, SQuAD 2.0, 80% of the Legal Text Retrieval Zalo 2021 training challenge dataset, and the dataset collected from thuvienphapluat.vn. Additionally, our synthetic query data generated for pre-training is also used in this fine-tuning process. For all datasets, we create hard negative passages for each training query from the BGE-M3 dense retrieval model.

For each query $q^{+}$, the positive passage $p^{+}$ forms a pair $(h_{p^{+}},h_{q^{+}})$. The negative samples $\{p^{-}\}$ included hard negatives identified by the BGE-M3 dense retrieval and in-batch negative passages. The training objective for both models is to maximize the similarity between the query and the positive passage while minimizing the similarity between the query and the negative passages. This is achieved using the InfoNCE loss function:

where $\tau$ is a temperature hyper-parameter fixed to 1, and $\text{sim}(\cdot,\cdot)$ represents the dot product similarity function.

Table 2 presents the datasets used for fine-tuning and evaluation. Our primary fine-tuning datasets include MS-MARCO [17], SQuAD 2.0 [21], 80% of the training set from the Legal Text Retrieval Zalo 2021 challenge (Legal Zalo 21), and the newly introduced TVPL dataset. Since MS-MARCO and SQuAD 2.0 are originally in English, we translated them into Vietnamese using Google Translate, following the approach of [20]. The use of large, translated datasets has proven beneficial for improving the performance of monolingual retriever models due to the absence of comparably large annotated datasets for Vietnamese.

We developed TVPL — a new benchmark dataset for Vietnamese legal text retrieval. TVPL is named after the thuvienphapluat.vn website, from which its legal QA articles were sourced. The dataset comprises a training set of 165,334 queries and a test set of 10,000 queries, along with a corpus of 224,006 legal passages. This dataset addresses previous limitations of scale and diversity, providing a more comprehensive benchmark for evaluating retrieval models in the Vietnamese legal domain.

Additionally, our 507,152 generated queries using the Llama3-70B model based on passages extracted from Vietnamese legal texts are also used in this fine-tuning stage. This synthetic dataset spans a broad range of legal domains, as shown in Figure 3.

For evaluation, we used 20% of the Legal Zalo 21 dataset and 10,000 test queries from TVPL for in-domain testing. Additionally, we employed the Vietnamese Wiki Question Answering dataset from the Zalo AI Challenge 2019 for out-of-domain evaluation.

We used the synthetic dataset for pre-training, with text segmented by the Underthesea library. During pre-training, we select a batch of passages at each step and randomly choose a candidate query as context for each passage to form a relevant pair. The encoder for CoT-MAE was initialized with a pre-trained PhoBERT-base-v2 model, while the decoder was trained from scratch. Pre-training was conducted for 2,000 steps using the AdamW optimizer with a learning rate of 1e-4, a batch size of 1024, and a linear schedule on a TPU v4-8. After training, the decoder was discarded, and only the encoder was retained for fine-tuning.

For fine-tuning, we utilized the four annotated datasets presented in the previous subsections along with our generated synthetic data. Legal Zalo 21 passages were chunked to PhoBERT’s 256-token limit. Hard negatives for each query were mined using the BGE-M3 dense retrieval model. The bi-encoder model was trained with a batch size of 64 across 4 TPU chips, optimized with AdamW at a 2e-5 learning rate for 170,000 steps (5 epochs), pairing each query with one positive passage and 7 hard negatives. ColBERT was trained with a batch size of 16, 15 hard negatives, and a longer training period of 290,000 steps (9 epochs), as the model continued improving throughout. ColBERT embeddings were compressed to 2 bits per dimension for evaluations.

For evaluation, our baseline methods include both sparse and dense retrieval approaches, as outlined in Table 3. The sparse retrieval baseline is represented by BM25. For dense retrieval baselines, we include results from the monolingual models vietnamese-sbert [10] and vietnamese-bi-encoder [20], as well as the multilingual models BGE-M3 [4] and $\text{mE5}_{\text{base}}$ [28].

As shown in Table 3, the results demonstrate that fine-tuning with additional generated data improves retrieval performance across all evaluation metrics. Pre-training further enhances these results, with notable improvements in both the bi-encoder and ColBERT models. ColBERT, with its more granular multi-vector retrieval, achieves the highest scores across both the TVPL and Legal Zalo 21 benchmarks.

To further evaluate the robustness and generalization capabilities of our models, we conducted an out-of-domain evaluation on the Vietnamese Wiki Question Answering dataset from the Zalo AI Challenge 2019. This dataset contains query-passage pairs covering a wide range of topics beyond the legal domain, on which our models were specifically pre-trained and fine-tuned.

Despite being trained solely in the legal context, our models demonstrated improved performance on the out-of-domain dataset, as shown in Table 4. Notably, the pre-trained and fine-tuned ColBERT model achieves scores close to those of larger multilingual retrievers, such as $\text{mE5}_{\text{base}}$ and BGE-M3. These models, with significantly larger parameter counts ($\text{mE5}_{\text{base}}$ at 270M and BGE-M3 at 560M, compared to our models with 124M), are still strong candidates for zero-shot retrieval tasks. The results suggest that our approach, though specialized for legal text, maintains strong generalization capabilities in broader retrieval scenarios.

We analyze the improvements brought by the aspect-guided query generation method (Section 3.3) compared to basic prompting, where the LLM generates queries directly from input passages. Performance is evaluated using passage hit rate (the percentage of queries retrieving their corresponding passage) and document hit rate (the percentage of queries retrieving the correct document). We use the BGE-M3 dense retriever [4] to rank the top-$k$ relevant passages for each query.

As shown in Table 5, the aspect-guided method significantly outperforms basic prompting across different top-$k$ values (10, 20, 40). At $k=10$, it achieves an 82.06% passage hit rate and 91.60% document hit rate, compared to 8.26% and 87.23%, respectively, for basic prompting.

We examine the impact of increasing the number of bits for compression on both retrieval accuracy and storage requirements. Experiments were conducted on 224,006 passages from the TVPL dataset, with evaluation metrics on its test set. ColBERT’s residual compression, as proposed in [23], offers improvements in both storage and retrieval performance. While higher bit sizes improve performance slightly, they come with a significant increase in storage. In contrast, The 1-bit ColBERT configuration performs competitively with minimal storage (647.33MB), even outperforming bi-encoder (672.02MB) in metrics like MRR@10 and MAP@10.