VLSP 2023 - LTER: A Summary of the Challenge on Legal Textual Entailment Recognition

Nguyen et al. (2023b) Their framework utilizes label models to ensemble predictions from two available large language models (LLMs). The first LLM is mT0, a multilingual LLM that is instruction-finetuned on a mixture of tasks. The second LLM is a Vietnamese Llama-2 model that is finetuned specifically for the task of legal text entailment. They prompt the two LLMs with a temperature variable greater than zero to obtain many provisional (often noisy) predictions. Afterward, they leverage label models, which help adjudicate predictions based on agreements and disagreements among them, to ensemble the noisy predictions.

Nguyen et al. (2023e) They proposed methods for this task involving three phases: translation, clause matching, and utilization of large language models. The translation process converts Vietnamese inputs into English inputs to utilize large language models excelling at processing English. Then, the clause-matching process starts with implementing regex techniques to extract clauses and their corresponding titles from the given articles. Then, BM25 Okapi is applied to calculate the relevant score of each clause which is considered relevant if its score is greater than a chosen threshold by parameter search. Finally, each statement and its relevant clauses are put into a designated template fed to an LLM, which emits a response indicating Yes or No.

Nguyen et al. (2023d) They proposed on top of a pre-trained language model a layer for fine-tuning based on Support Vector Machines Cortes and Vapnik (1995) kernel formulations. The proposed method leverages an intelligent combination of two popular kernel functions, polynomial kernel and Gaussian kernel, tightly integrated with the last layer of the pre-trained language model. They call this layer the “SVM layer” because it provides an innovative computational formula that efficiently processes data and enhances the representational capabilities of the model.

Ba et al. (2023) They adopt a transfer learning approach, focusing on transformer-based language models Vaswani et al. (2017) for semantic textual similarity tasks in Vietnamese legal text. They explore a range of pre-trained models applicable to Vietnamese. To enhance the models’ understanding of nuanced semantic relationships, they employ a contrastive learning approach. Specifically, they utilize the CrossEncoder architecture. The CrossEncoder processes pairs of texts simultaneously in a single forward pass, allowing it to capture intricate interactions between the claim and sentence comprehensively.

Cao et al. (2023) They utilized a pre-trained language model BERT (Bidirectional Encoder Representations from Transformers) specifically for Vietnam Legal Law (VNBert Law). Due to the limited training data, they added additional data from ALQAC 2021 and broke down the legal articles into smaller samples based on their individual items. To evaluate and label the data, they split the ”legal_passage” into sentences that had similar similarity ratings as determined by the BM25_Score with the corresponding statements. The pairs with the highest BM25_Score were considered as representing the best match between the ”legal_passage” and the ”statement”. The input format for the BERT-based model consisted of the following structure: [CLS] Statement [SEP] BM25_Score [SEP] Legal_passage [SEP].

It is to check if the system can have consistent predictions for related examples. Related examples are grouped by their (very high) similarity, for instance, a) ”Nguyễn Văn A sinh ngày 03/10/2003 thì ngày 02/10/2023 A đủ tuổi kết hôn” and b) ”Nguyễn Văn A sinh ngày 03/10/2003 thì ngày 03/10/2023 A đủ tuổi kết hôn”. As shown in Tables 4, when considering this, the accuracy drops substantially, which indicates that participants’ systems have difficulties in logically reasoning of the details. Among those participants, team NOWJ has a system with the best logical consistency (Tables 4 and 5).

It is one of the fundamental problems in natural language processing. If a system is built with content-similarity in mind, e.g., word co-occurrence frequency, a negated sentence is highly similar to its origin, thus indistinguishable by such a system. To deal with the negation problem, a system needs to have high logical consistency. To see this, we measure the accuracy of the systems on the subset of the test examples where we have pairs of a statement and its paired negated statement, where a system must provide correct answers for both statements of a pair to score. The results are shown in Table 6. The top-3 teams of overall accuracy also have the best performance on negation-pairs. Though, the accuracy in this assessment is only 0.6154, which means there is much more room for improvement. Even systems utilizing LLMs may still suffer from their limitations in dealing with negation Nguyen et al. (2023c).

Over 139 test examples, 29/139 examples were predicted correctly by all 5 teams, 1/139 example was predicted incorrectly by all 5 teams, and 107 examples were correctly predicted by at least 3/5 teams. When considering the logical consistency problem, this legal textual entailment recognition task is considerably challenging, where 21/139 examples were predicted correctly by all 5 teams, 12/139 example was predicted incorrectly by all 5 teams, and 71 examples were correctly predicted by at least 3/5 teams. Even though our dataset is created essentially based on the main text of the law documents, in concrete legal events, things, and their relationships are much more complicated, so there is still a lot of work to be done on the way, especially to deal with inherent challenges in utilizing LLMs for legal text processing Anh et al. (2023) including, for instance, interpretability, explainability, and bias mitigation.