Query-driven Relevant Paragraph Extraction from Legal Judgments

Legal professionals including lawyers, judges and paralegals, often need to sift through voluminous legal judgments that encompass crucial insights for case law interpretations and judicial reasoning. These judgments, often lengthy, contain nuanced paragraphs holding the key to understanding legal principles, precedents and arguments. Finding relevant case law accounts for roughly 15 hours per week for a lawyer Lastres (2015) or nearly 30% of their annual working hours Poje (2014). Recent advances in NLP offer new possibilities to bridge this gap by providing summaries of these documents (e.g., Bhattacharya et al. 2019; Shukla et al. 2022 inter alia). Nonetheless, practitioners still face challenges in navigating these texts to uncover specific paragraphs that address their queries. The current manual approach is labor-intensive and susceptible to overlooking essential details. Automating this process of identifying paragraphs relevant to the query streamlines legal research, allowing them to access relevant information efficiently.

Finding relevant paragraphs to a query is a challenging task unlike traditional adhoc information retrieval. Firstly, the legal domain is characterized by a vast and intricate vocabulary, interwoven with domain-specific jargon that can vary across different legal jurisdictions. This linguistic complexity demands an in-depth understanding of nuanced legal concepts, posing a substantial challenge for automated systems. The variation in legal writing style further compounds the challenge. Judgments may employ different degrees of formalism and offer varying levels of explicitness. These nuances can lead to difficulties in discerning context and accurately identifying relevant paragraphs that address specific queries. Another key challenge stems from the evolving nature of the legal case law. New legal doctrines, precedents and interpretations continually emerge, leading to an ever evolving array of legal concepts and principles. This dynamism necessitates a flexible and adaptive approach to comprehend new queries and determine relevance.

To investigate the ability of current retrieval models to identify relevant paragraphs, a high-quality labeled dataset is imperative. However, creating such datasets is resource-intensive, often necessitating the involvement of legal experts to produce queries and relevance labels. In this study, we employ distant supervision to construct a dataset tailored for the task of query-driven relevant paragraph extraction from legal judgments by the European Court of Human Rights (ECtHR) which addresses grievances by individuals against states for alleged violations of rights outlined in the European Convention of Human Rights. Our approach capitalizes on the case-law guides available through the ECtHR’s Knowledge Sharing platform¹¹1https://www.echr.coe.int/knowledge-sharing. We pose the case-law guide’s section headers as queries, mirroring the legal concepts professionals utilize when searching within ECtHR judgments. We gather relevance signals by identifying the pinpointed citations to the paragraphs in the judgments within these guides under each section. Further, we meticulously design various splits to assess the generalizability of systems towards new queries (legal concepts), adapting to the evolution of law.²²2Our dataset is made available at https://github.com/TUMLegalTech/ParagraphRetrievalECHR/

As a second contribution, we assess the performance of current retrieval models in a zero-shot manner using our dataset and further establish fine-tuning benchmarks employing diverse retrieval techniques encompassing dense bi-encoder and cross-encoder architectures. Our experiments reveal the drastic gap between fine-tuned and the zero-shot performance. Furthermore, we investigate into the efficacy of fine-tuning a general pre-trained model that was fine-tuned using other retrieval datasets (such as BERT fine-tuned on MSMARCO), comparing it against a legally pre-trained model (such as LegalBERT) that remains untouched by other retrieval datasets except ours. This investigation revealed that legal pre-training helps to handle distribution shift of the corpus, but still lacks in handling the distribution shift towards unseen queries.

While complete fine-tuning has shown better performance, the trend towards larger models with billions or trillions of trainable parameters makes this fine-tuning process resource-intensive and costly. This spurred the exploration of Parameter Efficient Fine-Tuning (PEFT) strategies which update only a small number of extra parameters while keeping the original pre-trained model parameters frozen. In our study, we delve into this emerging area by evaluating representative methods of PEFT, namely Adapter Houlsby et al. (2019), prefix-tuning Li and Liang (2021) and LoRA Hu et al. (2021), within the context of our paragraph retrieval dataset. This investigation contributes to the ongoing discourse regarding the practicality of adopting PEFT in the realm of Information Retrieval Pal et al. (2023); Tam et al. (2022); Ma et al. (2022); Jung et al. (2022). Our experiments demonstrate that PEFT methods achieve comparable performance to full fine-tuning on both seen and unseen queries, with the choice of the best PEFT method contingent on configuration such as general vs. legal pre-training and bi- vs. cross-encoder settings.

Our task of query-driven relevant paragraph extraction from legal judgements is defined as follows: Given a query $Q$ and a judgement document $J$ composed of $n$ paragraphs $P_{J}=\{p_{1},p_{2},\ldots,p_{n}\}$, the objective is to identify the subset of paragraphs $P^{+}_{J}\in P_{J}$ which are relevant to the query.

We benchmark our task of identifying relevant paragraphs from a legal judgement given a query using the following models. We compute relevance score for each paragraph in given judgement with respect to the query and obtain the top-k most relevant paragraphs with the highest scores.

Recently, Xiong et al. 2020 proposed Approximate nearest neighbor Negative Contrastive Learning (ANCE) mechanism for dense retrieval. Instead of random or static BM25 negatives, ANCE constructs negatives using the being-optimized dense retrieval model. This helps to align the distribution of negative samples based on the models’ training dynamics. While the model undergoes updates with each iteration, it would be expensive to update the negatives for every batch based on the updated model. Hence we asynchronously refresh the negatives at every checkpoint to reduce the computational cost to construct them.

These above methods follow a single-vector paradigm where each query and each paragraph is encoded into a single high-dimensional vector which is used to calculate relevance using a dot product. Khattab and Zaharia 2020 proposed a late interaction method named contextualized late interaction over BERT (ColBERT) where queries and documents are encoded at a finer granularity into multi-vector representations and relevance is estimated using interactions between these two sets of vectors. ColBERT produces an embedding for every token in the query and the paragraph and computes relevance as the sum of maximum similarities between each query vector and all vectors in the document as $rel(q,p)=\sum_{i=1}^{N}\max_{j=1}^{M}Q_{i}.D_{j}^{\intercal}$ where Q is a query encoding matrix corresponding to N token vectors and D denotes the paragraph encoding matrix corresponding to M token vectors.

Initially, we investigate the performance of retrieval models in a zero-shot evaluation scenario, where models trained on the MS MARCO paragraph ranking dataset Bajaj et al. (2016) - a large-scale adhoc retrieval dataset derived from the Bing search log containing 8.8 million passages and around 800K queries for training, are directly evaluated on our legal judgement paragraph ranking dataset. We examine the following models: (i) DPR⁸⁸8https://huggingface.co/facebook/dpr-question_encoder-multiset-base (ii) ANCE⁹⁹9https://huggingface.co/sentence-transformers/msmarco-roberta-base-ance-firstp (iii) ColBERT ¹⁰¹⁰10https://github.com/stanford-futuredata/ColBERT (iv) Cross encoder¹¹¹¹11https://huggingface.co/cross-encoder/ms-marco-MiniLM-L-12-v2. We also evaluate a legal-domain-specific encoder model, LegalBERT Chalkidis et al. (2020) which is pre-trained on diverse English legal texts encompassing legislative content, court cases, and contracts using cosine similarity between obtained [CLS] embeddings as relevance score. Notably, LegalBERT has been exposed to case law from ECtHR.

Subsequently, we fine-tune these models on the training split of our legal judgment paragraph extraction dataset. We create two variants of each model, with distinct initializations: (i) model already fine-tuned on MSMARCO (models used in the zero-shot evaluation) and (ii) LegalBERT.

PEFT aims to tune only a small portion of parameters rather than the full parameters as in traditional fine-tuning. PEFT approaches fall into three primary categories: Parameter Composition, Input Composition, and Function Composition Ruder et al. (2022). Given a neural network $f_{\theta}:X\to Y$, it is decomposed into a sequence of functions $f_{\theta}=f_{\theta_{1}}\odot f_{\theta_{2}}\odot\ldots f_{\theta_{l}}$, where $\theta_{1},\theta_{2},\ldots,\theta_{l}$ represent parameters which are held constant in PEFT and a module with parameters $\phi$ is introduced, which are updated during training to modify the $i^{th}$ sub-function as follows: Parameter composition involves interpolating models’ parameter with new parameters as $f^{\prime}_{i}(x)=f_{\theta_{i}\odot\phi}(x)$. Input Composition augments a model’s input with a learnable parameter vector as $f^{\prime}_{i}(x)=f_{\theta_{i}}([x,\phi])$ Function composition augments a model’s functions with new task-specific functions as $f^{\prime}_{i}(x)=f_{\theta_{i}}\odot f_{\phi}(x)$. We pick one representative method from each category and study their performance on our retrieval task.

We investigate the effect of PEFT by applying each method separately on bi-encoder and cross-encoder, using MSMARCO and LegalBERT initializations. Among bi-encoders, we choose COLBERT due to its better performance in full fine-training. We report Recall@k% for $k=\{2,5,10\}$ in Table 2.

We present an empirical study focused on the task of extracting relevant paragraphs from legal judgments based on the query. We rigorously curate a dataset for this task from ECtHR jurisdiction, leveraging the case-law guides produced by the court’s registry. We assess the current retrieval models on this task in a zero-shot way to emphasize the need of retrieval specific pre-training objectives. We further fine-tune several models encompassing bi- and cross-encoders for this task. We evaluate the generalizability of different fine-tuning models when faced with unseen concepts or queries to illustrate how legal pre-training can effectively address distribution shifts on the corpus side but still faces challenges in adapting to shift on the query side. In addition, we demonstrate the efficacy of different PEFT methods on these retrieval methods shedding light on their intricate effects concerning legal pre-training, bi-encoder, and cross-encoder models. Our findings reveal that there is no one-size-fits-all PEFT method that performs well across all settings. We hope that both our dataset and the fine-tuned models will be useful to the research community working in the space of legal information retrieval.

In this study, we treat each paragraph as an independent unit during the training of neural models. However, it’s important to note that paragraphs are not entirely independent; they often constitute small excerpts from longer documents, and their content may not always provide a comprehensive estimate of their relevance. To be more precise, some paragraphs draw context not just from other paragraphs within the same document but also from other documents, as evident through citations and cross-references to other judgments or texts within each paragraph. In the future, harnessing this inter-paragraph and cross-document contextual signal could lead to a more enriched understanding of each paragraph’s relevance.

Furthermore, this practice of segmenting documents into smaller chunks is common in retrieval tasks, where documents are broken down into shorter lengths for indexing in retrieval systems. This may lead to losing some of their context in the process. It’s worth noting that this chunking effect could be more pronounced in our task compared to more fact-focused general retrieval tasks. As a result, future research should explore methods to effectively capture contextual information from the sequential nature of paragraphs within documents and develop discourse-aware representations.

Most retriever systems follow a two-stage pipeline approach, where a pre-fetcher first aims to return all relevant ones followed by a re-ranker which attempts to make more relevant ones appear before less relevant ones. In this work, we explicitly focused our experiments on the pre-fetcher component leaving the second component for future work.

A specific challenge with respect to PEFT methods are that they converge slower and relatively more sensitive to hyper-parameters such as learning rate than full fine-tuning. We do observe same characteristics during our work and have to bypass the problem by training for more epochs and experimenting with different hyper-parameters. It is thus imperative to analyse them more theoretically and design more robust and stable training strategies for PEFT methods in the future.

Additionally, while our findings of the relevant paragraph retrieval experiments are specific to the ECtHR domain and datasets, comparable experiments in other domains will see variation based on the nature of the legal concepts and the legal documents. Nevertheless, all derivation of insights from legal case data comes with jurisdiction-related limitations.

With the release of our data as a public resource, we do not foresee any ethical concerns in a repurposed and bundled release of this dataset as both the judgements data and the caselaw guides are already available through the HUDOC and ECHR KS platform respectively and it complies with the ECtHR data policy. These decisions, although not anonymized, include the real names of individuals involved. However, our work does not engage with the data in a way that we consider harmful beyond this availability. We believe that this work can foster further research in this data scarce legal NLP field, to build assistive technology for legal professionals. We are conscious by employing pre-trained language models, we inherit the biases they may have acquired from their training corpus and need to be further scrutinized of any biases that may arise and is crucial to ensure that the systems developed are fair.