ALJP: An Arabic Legal Judgment Prediction in Personal Status Cases Using Machine Learning Models

With the development of artificial intelligence (AI) and judicial informatization reform, the development of intelligent justice has received a great deal of attention. The proper application of artificial intelligence technology can improve judicial practitioner efficiency, optimize justice methods, and reduce sentence inconsistency. Several intelligent solutions, such as intelligent legal document generation, automatic question answering, and automatic speech recognition in the court system, have been effectively implemented in casework. Legal judgment prediction (LJP) is the critical point of artificial judicial intelligence. LJP systems aim to predict judgment results according to the facts of cases with feasible judgment suggestions, such as the prediction of charges, imprisonment terms, and applicable law articles. LJP systems can also assist litigants, attorneys, students, and teachers by improving their work and time efficiency while reducing the risk of making mistakes, so it serves as a valuable resource for professionals.

Being in the legal field is a boatload of predicting and the risk of facing various hidden dangers while doing so. When deciding whether or not to take a new case, an astute attorney would evaluate the nature of the legal issues at hand and the case’s likely results. Typically, wise attorneys would avoid making precise, unambiguous predictions about what they likely believe the conclusion will be. They safely distanced themselves from a seemingly ironclad prediction, emphasizing that they do not guarantee the legal results. New attorneys often make bold predictions; they almost end up on the wrong side of a displeased client who later remembers how confidently boastful the first prediction was, especially if the case goes astray from the anticipated outcome. The advent of Artificial Intelligence (AI) in the legal industry enhances attorneys’ prediction abilities. It becomes more practical to analyze a huge corpus of legal cases and generate predictions for a newly given legal case utilizing AI techniques, such as Natural Language Processing and Machine Learning. Using AI and deep learning in LJP is beneficial in providing fast decision-making and outcomes which allows input data to be instantly verified, unbiased judgment towards gender or nationality, analysis with previous instances of comparable patterns, and identify situations with substantial variation in human and AI choices makes it easier to uncover corruption.

Several studies have investigated the possibility of predicting judgment outcomes in legal cases written in English, Hindi, and Chinese. No study has investigated the prediction of legal judgment in Arabic, especially in personal status cases such as custody, divorce, and annulment of marriages. Thus, we develop an LJP system that performs two tasks in this paper. The first task is to predict judgment decisions and relevant law articles or evidence in (custody and annulment of marriage) using the pleading of the case. The second is to predict the probability of possible outcomes of personal status (custody and annulment of marriage) cases given the plaintiff’s claim and defend answer. We developed an Arabic legal prediction dataset for personal status cases (using the Kingdom of Saudi Arabia (KSA) as a case study). The dataset was generated using a sample collection published Ministry of Justice and a simulated dataset by experts in the domain. We evaluated the proposed model using different machines and deep learning models such as SVM, LR, LSTM, and BiLSTM with different data representation techniques such as TF-IDF and word2vec on the developed dataset.

Several studies have utilized machine learning techniques to predict judicial outcomes. Aletras et al. [1] presented the first comprehensive research of predicting the result of cases heard by the European Court of Human Rights (ECHR) based only on the textual content presented in the court to determine if a human rights article has been violated or not. An Support Vector Machine (SVM) classifier trained on textual information using N-grams and topics in Human Rights Documentation cases was used, achieving high accuracy in predicting court outcomes (79%). Sil and Roy [2] also proposed a model which aims to deliver justice by providing judicial argument-based analysis using the SVM algorithm. The model is trained on features like years of marriage, dowry details, and postmortem reports from a dataset of ‘dowry death’ cases in West Bengal. The model achieved 93% accuracy in binary classification. Medvedeva et al. [3] proposed a machine learning model to predict court decisions using textual information from ECHR cases. The program uses SVM Linear classifier and word n-gram TF-IDF to analyze textual data, predicting 75% of cases correctly. Similarly, Shaikh et al. [4] proposed a machine learning model to predict outcomes of murder-related cases in the Delhi District Court. Several machine learning classifiers were evaluated in this task, including Classification and Regression Trees (CART), Bagging, Random Forest, and SVM, to predict trial outcomes using several handcrafted features obtained by manually analyzing the cases. The results show that the best performance is obtained using Bagging, RF, and SVM, CART with ranges between 91,86% and 90.70%.

Several studies have utilized deep learning techniques in predicting legal case outcomes. Zhang et al. [5] proposed an automatic law article prediction model based on Deep Pyramid Convolutional Neural Network. They predict the relevant law article for cases using case description and associated legal provisions. The results show that the proposed method outperforms various state-of-the-art baselines models on several public datasets [6]. Li et al. [7] proposed a neural network model based on an element-driven attention mechanism that takes the textual description of a criminal case as the input and predicts the charges, applicable law articles, and prison terms. The approach is evaluated on a real-world dataset containing $125,830$ judgment documents of criminal cases published by China Judgments. The model accuracy for element prediction, charge prediction result, law article prediction, and prison term was 98.83%, 97.92%, 98.16%, and 82.13%, respectively. Most existing methods follow the text classification framework that fails to model the complex interactions among complementary case materials. Long et al. [8] formalized the task as Legal Reading Comprehension according to the legal scenario. The framework predicts the final judgment results based on three types of information, including fact description, plaintiffs’ pleas, and law articles and predict if a certain plea in a given civil case would be supported or rejected.

Multi-task learning (MTL) models were also examined for LJP in several studies. The MTL model considers the relationship between subtasks such as law articles, charges, and penalty terms in LJP task. Zhong et al. [9] proposed a topological multi-task learning framework that formulates the dependencies among subtasks (law articles, charges, fines, and the term of penalty) as directed acyclic graph (DAG) to jointly predict the trial subtasks outcomes. Their approach outperforms single-task baselines and conventional multi-task learning models on three Chinese criminal case datasets. Wang and Jin [10] proposed a MTL LJP model based on CNN-BiGRU and is used to improve the accuracy and efficiency of legal judgment prediction. Several data representation were used including TF-IDF and word2vec, word embedding, fact encoding, and document representation achieving accuracy of $95.1\%$ for Law articles prediction, $95.2\%$ for Charges, and $72.6\%$ for the term of penalty. Huang and Lin [11] built a multi-task deep neural network classification model based on integrating CNN with the attention LSTM model to achieve high precision of crime and related law prediction. The model is evaluated on a dataset of $626,600$ judicial documents collected from the Internet achieving an average F1 score of 93.62% and 90.84% for crime, related law prediction, respectively.

Li et al. [12] proposed a multi-channel attentive neural network that uses attention mechanism and BiGRU hierarchical sequence encoder to learn better semantic representation and interaction among different parts of case descriptions. The multi-channel attentive encoders take a generated law articles based on the case fact description and defendant persona as input and pass them into three hierarchical encoders (fact-channel encoder, persona channel encoder, article-channel encoder) incorporating with word-level and sentence-level attention context vector to predict the charges and prison term. Yao et al. [13] proposed a novel gated hierarchical multitask learning network to jointly model multiple sub-tasks (law article, charge, and term of penalty) in judicial decision prediction. The model combines a Gated Hierarchical Encoder (GHE) to extract in-depth semantic information of fact description from multiple perspectives and a Dependencies Auto-learning Predictor (DAP) to learn the dependencies among sub-tasks dynamically. The proposed model takes the fact description as input and predicts law article, charge, and term of penalty.

The attention mechanism has been successfully used in many NLP tasks in recent years. A number of studies used attention mechanisms in the task of predicting judicial outcomes. Bao et al. [14] proposed an attention neural network that uses relevant articles to improve the performance and interpretability of charge prediction tasks. The model uses the fact description to extract relevant law articles that assist in locating key information from the fact description and help improve the performance of charge prediction. To address the challenges of predicting judgment in lengthy cases, Sukanya and J.Priyadarshini [15] presented an effective hierarchical attention deep neural network model with fine-tuned transformer to predict legal cases outcomes. Yang et al. [16] employed LSTM with self-attention to simulate a judge’s recurrent reading behavior utilizing semantic mutual information between evidence and article. Xu et al. [17] presented an end-to-end model to solve the task of LJP. To distinguish confusing charges, they propose a novel graph neural network to automatically learn subtle differences between confusing law articles by capturing essential but rare features and design a novel attention mechanism that fully exploits the learned differences to extract compelling discriminative features from fact descriptions attentively.

Kowsrihawat et al. [18] proposed a prediction model of criminal cases using End-to-End BiGRU deep learning neural networks. Their model imitates a process of legal interpretation, whereby recurrent neural networks read the fact from an input case and compare them against relevant legal provisions with the attention mechanism. Some related studies have combined several deep neural models.  Yuan et al. [19] proposed a framework for the automated judging based on ensemble strategy that combines many deep neural network models and manual law features to solve the problem of data imbalanced. Simultaneously, they increase the framework’s performance by enhancing the data. Their model shows that data enhancement and ensemble strategy can improve the performance of judgment prediction.

Transformer-based models have shown a tremendous impact on many NLP problems. Similar to other NLP problems, transformer-based models were successfully used in predicting the outcome of the legal cases. Wang et al. [20] utilized the recently widely used pre-trained language model Bidirectional Encoder Representations from Transformers (BERT) for LJP. BERT model significantly improves accusation prediction accuracy compared to other deep learning models with Word2Vec representation. Similarly, Chalkidis et al. [21] developed a hierarchical version of BERT for judgment prediction. They present a new publicly available dataset of English legal judgment prediction cases from the ECHR. Zhu et al. [22] proposed a Transformer-Hierarchical-Attention Multi-Extra Network that takes fact description, court view, and basic information of the defendant as input to predict law articles, charges, and terms of penalty. The dataset they used is CJO consists of criminal cases published by the Chinese government from China Judgment Online.

Some studies have employed Tensor decomposition techniques in legal cases outcome prediction. Guo et al. [23] developed a new algorithm based on innovative tensor decomposition and ridge regression for judgment prediction. The model tested on a dataset obtained from the Chinese Referee Document Network.In the same vein, Guo et al. [24] build an intelligent judgment approach, which is based on the relationship-driven recurrent neural network and restricted tensor decomposition. The recurrent neural network were used for intelligent judgment of multiple accusations in legal cases. The model tested on legal cases obtained from a Chinese refereeing study network.

Previous studies in LJP have utilized various machine and deep learning techniques, including SVM, LR, and K-Nearest Neighbors. Most studies utilized deep learning techniques and sequence models, while some used attention mechanisms for predicting legal case outcomes. Tensor decomposition-based models were also employed. LJP models rely on law articles and predict law articles, charges, and penalties. Researchers have formulated problems as single-task learning or multi-task learning, predicting single or multiple outcomes. Previous studies were mainly conducted in Chinese, Hindi, and English, and mostly focused on crime cases. However, no studies have been conducted in Arabic or personal status cases, as personal status cases in KSA are adjudicated at the judge’s discretion, not on law articles.

Several dataset were used in the filed of LJP. These dataset were generally written in Chinese, English, and Hindi. To serve the purpose of this study, we developed an Arabic LJP dataset for Saudi Arabian Personal Status Cases, we have first collected personal status cases published by the ministry of justice and we also generated a new sample of cases throughout experts in the field. Table 1 shows information about the Ministry of Justice dataset. Each article in the sample collection contains the topic which provide details of case category, the evidence which provide details of the reasons and legal evidence that was relied upon in judging the case, a summary of the case which include a summary of the claim, answer, and judgment, the final judgment which is a description of the claim, answer, and sessions that took place in the court, in addition to the judgment on the case and the. The sample collection has five type of cases which are custody, annulment of marriage, visiting, divorce, and alimony. We have only consider in this research custody and annulment of marriage cases due to limitation and variation of outcomes of the rest of the personal status cases in the current sample of collection. The dataset has a total number of $49$ cases that each has a true judgment outcome.

The dataset textual content were obtained from portable document format(PDF) files and many words are extracted incorrectly due to the font in which the data was written. Thus, to correct some of the extracted data, several data cleaning and manual corrections were performed.

We have formulated the problem as a multi class classification problem. The multi class classification task, takes a pleading text as input and generate the judgment (one of multiple outcomes) and reasons/evidences (one of multiple reasons) as output. The original trail cases were analyzed with experts in the law field (lawyers and professors) and a list of possible and more common judgement decisions were obtained. Based on this analysis, we formulated the problem of predicting judgement in custody cases as a multi-class classification problem of four classes, namely, mother grant full custody, father grant full custody, children over seven years old have parental choice, while those under seven have custody with their mother until they reach seven, and other. Table 2 shows the judgment classes in custody cases. For the reasons or law article prediction, we formulated the problem as a multi class classification with eight classes correspond to the law articles used in the custody cases. For annulment of marriage cases, we had also four classes which are annulment of marriage with compensation, annulment of marriage without compensation, deny annulment, other. Table 3 show the classes of annulment of marriage. The reasons or law article prediction in the annulment of marriage cases were formulated as a multi class classification problem with 11 classes.

Table 4 shows an example of the binary classification problem.

To increase the model performance, another simulated dataset was collected throughout experts in the law domain, the experts are selected among Master and PhD students, and validated by Arabic speaking law professors. The dataset has a larger number of custody and annulment of marriage cases. As shown in Table 5 the simulated dataset consist of the plaintiff’s claim, the defendant’s response, pleading (a description of the sessions that took place in court), the judgment, and the evidences (the reasons and legal evidence that was relied upon in judging the case).

Several researchers have developed techniques for predicting the outcomes of cases in the legal profession. However, none of the proposed techniques were implemented in Arabic, and only a few attempts were implemented in English. This project aims to develop an Arabic legal judgment prediction system that utilizes deep learning models, such as LSTM and BiLSTM, and machine learning techniques, such as SVM and LR, to predict the judgment outcome and law articles from Arabic language case scripts, especially personal status cases in Saudi Arabia. Thus, we developed an Arabic LJP dataset from publicly available sample collections combined with the data artificially generated by law experts. The dataset was then analyzed to formulate the LJP task as a multi-class classification problem. We investigated several machine learning and deep learning models with different types of word representation. The results indicate that for predicting the judgment of the custody cases from pleading text, the SVM model with word2vec data representation achieved the highest accuracy of 88%. For the annulment of marriage cases, the LR model with TFIDF achieved the best result with 78% accuracy. For predicting the law article or evidence of custody cases, BiLSTM with TFIDF gave 25% accuracy, while for the annulment of marriage cases, the LR model with word2vec achieved 50% accuracy. For predicting the most probable judgment from claim and answer, the highest accuracy of 88% was obtained using SVM and LR with word2vec representation in custody cases and 68% using BiLSTM with TF-IDF in the annulment of marriage cases.