Multi-Defendant Legal Judgment Prediction via Hierarchical Reasoning

Legal judgment prediction has been studied in various jurisdictions (Zhong et al., 2020; Feng et al., 2022a; Katz et al., 2017; Chalkidis et al., 2019; Sulea et al., 2017a, b; Malik et al., 2021; Paul et al., 2020; Niklaus et al., 2021). Early studies on LJP focus on rule-based methods (Kort, 1957; Nagel, 1963; Segal, 1984) and machine learning algorithms (Aletras et al., 2016; Sulea et al., 2017a, b; Katz et al., 2017). Recent neural-based approaches jointly predict judgment results (law articles, charges and term of penalty) for single-defendant cases by modeling dependency between LJP subtasks (Zhong et al., 2018; Yang et al., 2019; Dong and Niu, 2021; Huang et al., 2021), leveraging legal knowledge (Hu et al., 2018; Gan et al., 2021; Yue et al., 2021; Ma et al., 2021; Lyu et al., 2022; Feng et al., 2022b), exploiting label information (Luo et al., 2017; Wang et al., 2019; Xu et al., 2020; Le et al., 2022; Liu et al., 2022; Zhang et al., 2023a), or employing pre-trained language models (Chalkidis et al., 2020, 2021; Xiao et al., 2021a). For multi-defendant cases, MAMD Pan et al. (2019) utilizes multi-scale attention to distinguish confusing fact descriptions of different defendants for multi-defendant charge prediction.

However, existing single-defendant LJP methods neglect the complex interactions among multiple defendants. Moreover, compared with the multi-defendant LJP method MAMD Pan et al. (2019), we follow the human judgment process to model hierarchical reasoning chains to distinguish different judgment results of various defendants and make accurate judgments for each defendant.

Multi-step reasoning by training or fine-tuning language models to generate intermediate steps has been shown to improve performance (Zaidan et al., 2007; Talmor et al., 2020; Yao et al., 2021; Hase and Bansal, 2021; Zhang et al., 2023b; Gu et al., 2021). Ling et al. (2017) generate natural language intermediate steps to address math word problems. Camburu et al. (2018) extend the natural language inference dataset with human-annotated natural language explanations of the entailment relations. Rajani et al. (2019) generate rationales that explain model predictions for commonsense question-answering tasks (Talmor et al., 2019). Hendrycks et al. (2021) finetune pre-trained language models to solve competition mathematics problems by generating multi-step solutions. Nye et al. (2021) train language models to predict the final outputs of programs by predicting intermediate computational results. Recently, Wei et al. (2022) propose chain of thought prompting, which feed large language models with step-by-step reasoning examples without fine-tuning to improve model performance.

However, these methods are not designed for more realistic legal reasoning applications (Huang and Chang, 2022). In this paper, we aim to use generative language models to capture hierarchical reasoning chains for the multi-defendant LJP task.

To the best of our knowledge, existing LJP datasets only focus on single-defendant cases or charge prediction for multi-defendant cases. Thus, we construct a Multi-defendant Legal Judgment Prediction (MultiLJP) dataset from the published legal documents in China Judgements Online²²2https://wenshu.court.gov.cn/. Instead of extracting labels using regular expressions as in existing works Xiao et al. (2018); Pan et al. (2019), we hire eight professional annotators to manually produce law articles, charges, terms of penalty, criminal relationships, and sentencing circumstances for each defendant in multi-defendant cases. The annotators are native Chinese speakers who have passed China’s Unified Qualification Exam for Legal Professionals. All data is evaluated by two annotators repeatedly to eliminate bias. Since second-instance cases and retrial cases are too complicated, we only retain first-instance cases. Besides, we anonymize sensitive information (e.g., name, location, etc.) for multi-defendant cases to avoid potential risks of structured social biases Pitoura et al. (2017) and protect personal privacy. After preprocessing and manual annotation, the MultiLJP consists of 23,717 multi-defendant cases. The statistics information of dataset MultiLJP can be found in Table 1.

We first formulate the multi-defendant LJP task. The fact description of a multi-defendant case can be seen as a word sequence $\mathbf{x}=\{w_{1},w_{2},...,w_{n}\}$, where n represents the number of words. Each multi-defendant case has a set of defendant names $E=\{\mathbf{e}_{1},\mathbf{e}_{2},...,\mathbf{e}_{|E|}\}$, where each name is a sequence of words $\mathbf{e}=\{w_{1},w_{2},..,w_{|\mathbf{e}|}\}$. Given the fact description $\mathbf{x}$ and the defendant name $\mathbf{e}$ of a multi-defendant case, the multi-defendant task aims to predict the judgment results of multiple applicable law articles, multiple charges, and a term of penalty. The law article prediction and the charge prediction subtasks are multi-label classification problems, and the term of penalty prediction subtask is a multi-class classification problem.

We introduce the criminal relationship and the sentencing circumstance as intermediate tasks to model the hierarchical reasoning chains for multi-defendant LJP and improve the prediction of the main judgment results. Criminal relationships refer to the relationships between defendants, specifically whether one defendant assisted other co-defendants during the commission of the crime. Sentencing circumstances refer to specific behaviors (such as confession and recidivist) or factors (like accessory and blind) that can influence the severity or leniency of a penalty³³3Details of criminal relationships and sentencing circumstances definitions together with their explanations can be found in Appendix A and B.. These two tasks are also multi-label classification problems. We denote the labels of law articles, charges, terms of penalty, criminal relationships, and sentencing circumstances as word sequences $\mathbf{y}^{l},\mathbf{y}^{c},\mathbf{y}^{t},\mathbf{y}^{r}$ and $\mathbf{y}^{s}$ respectively in this paper.

From the perspective of Sequence-to-Sequence (Seq2Seq) generation, each task can be modeled as finding an optimal label sequence $\mathbf{y}$ that maximizes the conditional probability based on the fact description, a specific defendant name and a specific task description $p(\mathbf{y}|\mathbf{x},\mathbf{e},\mathbf{d})$, which is calculated as follows:

$$p(\mathbf{y}|\mathbf{x},\mathbf{e},\mathbf{d})=\prod_{i=1}^{m}p(y_{i}|y_{1},y_{2},...,y_{m-1},\mathbf{x},\mathbf{e},\mathbf{d}),$$

(1)

where $m$ denotes the length of the label sequence, and the specific task description $\mathbf{d}$ is a semantic prompt that allows Seq2Seq generation models to execute the desired task. To accomplish the Seq2Seq generation tasks, we apply the Seq2Seq language model mT5 Xue et al. (2021) to generate label sequences as follows:

$$\mathbf{\hat{y}}=DEC(ENC(\mathbf{x},\mathbf{e},\mathbf{d})),$$

(2)

where $ENC$ refers to the encoder of the language model, $DEC$ denotes the decoder of the language model, and $\mathbf{\hat{y}}$ is prediction results composed of words. We use special [SEP] tokens to separate the different information to form the input of the encoder.

To distinguish different judgment results among various defendants, our method HRN follows hierarchical reasoning chains to determine each defendant’s criminal relationships, sentencing circumstances, law articles, charges, and terms of penalty. As shown in Figure 2, the hierarchical reasoning chains consist of two levels:

The first-level reasoning is for intermediate tasks. The first-level reasoning chain is to first identify relationships between defendants based on the fact description, the names of all defendants and the criminal relationship prediction task description $d^{r}$ as follows:

$$\mathbf{\hat{y}^{r}}=DEC(ENC(\mathbf{x},E,\mathbf{d}^{r})).$$

(3)

Then, we determine sentencing circumstances for the defendant $\mathbf{e}$ based on the fact description, name of the defendant $\mathbf{e}$, prediction results of criminal relationships and the sentencing circumstance prediction task description $\mathbf{d}^{s}$, that is:

$$\mathbf{\hat{y}^{s}}=DEC(ENC(\mathbf{x},\mathbf{e},\mathbf{\hat{y}}^{r},\mathbf{d}^{s}).$$

(4)

The second-level reasoning is for judgment prediction tasks. The second-level reasoning chain consists of a forward prediction process and a backward verification process. The forward prediction process is to predict law articles, charges, and terms of penalty (in that order) based on the fact description, the name of defendant $\mathbf{e}$, first-level reasoning results, and the forward prediction task description $\mathbf{d}^{lct}$ as follows:

$$\mathbf{\hat{y}^{lct}}=DEC(ENC(\mathbf{x},\mathbf{e},\mathbf{\hat{y}}^{r},\mathbf{\hat{y}}^{s},\mathbf{d}^{lct})).$$

(5)

Then, the backward verification process is to verify these judgment results in reverse order based on the fact description, the name of defendant $\mathbf{e}$, first-level reasoning results and the backward verification task description $\mathbf{d}^{tcl}$, that is:

$$\mathbf{\hat{y}^{tcl}}=DEC(ENC(\mathbf{x},\mathbf{e},\mathbf{\hat{y}}^{r},\mathbf{\hat{y}}^{s},\mathbf{d}^{tcl})).$$

(6)

To handle multi-defendant fact descriptions whose average length exceeds the length limit of the encoder, we adopt Fusion-in-Decoder (FID) Izacard and Grave (2021) to encode multiple paragraphs split from a fact description. We first split the fact description $\mathbf{x}$ into $K$ paragraphs containing $M$ words. Then, we combine multiple paragraph representations from the encoder, the decoder generates prediction results by attending to multiple paragraph representations as follows:

$$\mathbf{\hat{y}}=DEC(\mathbf{h}_{1},\mathbf{h}_{2},...,\mathbf{h}_{K}),$$

(7)

where $\mathbf{h}_{i}$ denotes the representation of the $i$-th paragraph of the fact description $\mathbf{x}$. Since all tasks are formulated as sequence-to-sequence generation tasks, we follow Raffel et al. (2020) to train the model by standard maximum likelihood and calculate the cross-entropy loss for each task. The overall loss function is formally computed as:

$$\mathcal{L}=\lambda_{r}\mathcal{L}_{r}+\lambda_{s}\mathcal{L}_{s}+\lambda_{lct}\mathcal{L}_{lct}+\lambda_{tcl}\mathcal{L}_{tcl},$$

(8)

where hyperparameters $\lambda$ determine the trade-off between all subtask losses. $\mathcal{L}_{r}$, $\mathcal{L}_{s}$, $\mathcal{L}_{lct}$ and $\mathcal{L}_{tcl}$ denote the cross-entropy losses of the criminal relationship prediction task, the sentencing circumstance prediction task, the forward prediction process and the backward verification process, respectively. At test time, we apply greedy decoding to generate forward and backward prediction results. Finally, the chain with the highest confidence is chosen for the final prediction.

We aim to answer the following research questions with our experiments: (RQ1) How does our proposed method, HRN, perform on multi-defendant LJP cases? (RQ2) How do the different levels of reasoning chains affect the performances of HRN on multi-defendant LJP?

To verify the effectiveness of our method HRN on multi-defendant LJP, we compare it with a variety of methods, which can be summarized in the following three groups:

• •

  Single-defendant LJP methods, including Topjudge (Zhong et al., 2018), which is a topological dependency learning framework for single-defendant LJP and formalizes the explicit dependencies over subtasks as a directed acyclic graph; MPBFN (Yang et al., 2019), which is a single-defendant LJP method and utilizes forward and backward dependencies among multiple LJP subtasks; LADAN (Xu et al., 2020), which is a graph neural network based method that automatically captures subtle differences among confusing law articles; NeurJudge (Yue et al., 2021), which utilizes the results of intermediate subtasks to separate the fact statement into different circumstances and exploits them to make the predictions of other subtasks.

• •

  Pre-trained language models, including BERT Cui et al. (2021), which is a Transformer-based method that is pre-trained on Chinese Wikipedia documents; mT5 Xue et al. (2021), which is a multilingual model pre-trained by converting several language tasks into “text-to-text” tasks and pre-trained on Chinese datasets; Lawformer Xiao et al. (2021b), which is a Transformer-based method that is pre-trained on large-scale Chinese legal long case documents.

• •

  Multi-defendant charge prediction method, including MAMD Pan et al. (2019), which is a multi-defendant charge prediction method that leverages multi-scale attention to recognize fact descriptions for different defendants.

We adapt single-defendant LJP methods to multi-defendant LJP by concatenating a defendant’s name and a fact description as input and training models to predict judgment results. However, we exclude some state-of-the-art single-defendant approaches unsuitable for multi-defendant settings. Few-shot Hu et al. (2018), EPM Feng et al. (2022b), and CEEN Lyu et al. (2022) annotate extra attributes for single-defendant datasets, not easily transferable to MultiLJP. Also, CTM Liu et al. (2022) and CL4LJP Zhang et al. (2023a) design specific sampling strategies for contrastive learning of single-defendant cases, hard to generalize to multi-defendant cases.

To accommodate the length of multi-defendant fact descriptions, we set the maximum fact length to 2304. Due to the constraints of the model input, BERT’s input length is limited to 512. For training, we employed the AdamW Loshchilov and Hutter (2019) optimizer and used a linear learning rate schedule with warmup. The warmup ratio was set to 0.01, and the maximum learning rate was set to $1\cdot 10^{-3}.$ We set the batch size as 128 and adopt the gradient accumulation strategy. All models are trained for a maximum of 20 epochs. The model that performs best on the validation set is selected. For the hyperparameters, $\lambda$, in the loss function, the best setting is {0.6, 0.8, 1,4, 1.2} for {$\lambda_{r}$, $\lambda_{s}$, $\lambda_{lct}$, $\lambda_{tcl}$}. Additionally, we set the number of paragraphs $K$, the number of words per paragraph $M$, and the output length to 3, 768, and 64, respectively. For performance evaluation, we employ four metrics: accuracy (Acc.), Macro-Precision (MP), Macro-Recall (MR), and Macro-F1 (F1). All experiments are conducted on one RTX3090.

Table 2 shows the evaluation results on the multi-defendant LJP subtasks. Generally, HRN achieves the best performance in terms of all metrics for all multi-defendant LJP subtasks. Based on the results, we have three main observations:

• •

  Compared with state-of-art single-defendant LJP methods, e.g., Topjudge, MPBFN, LADAN, and NeurJudge, our method HRN consider hierarchical reasoning chains and thus achieve significant improvements. Since the single-defendant methods do not consider criminal relationships and sentencing circumstances, they can not distinguish different judgment results among various defendants well. It indicates the importance of following the hierarchical reasoning chains to predict criminal relationships and sentencing circumstances for multi-defendant LJP.

• •

  As Table 2 shows, our method HRN achieves considerable performance improvements on all subtasks of multi-defendant LJP compared to pre-trained models BERT and Lawformer. This shows that modeling the hierarchical reasoning chains during the fine-tuning stage is crucial. Moreover, HRN, which combines mT5 and the hierarchical reasoning chains, significantly outperforms mT5, indicating that the language knowledge from pre-training and the information of the hierarchical reasoning chains are complementary to each other.

• •

  Compared to MAMD designed for multi-defendant charge prediction, our method HRN outperforms MAMD on charge prediction task. This shows the effectiveness and robustness of modeling reasoning chains in real-world application scenarios.

• •

  We employ gold annotations of criminal relationships and sentencing circumstances, rather than predicted ones, to enhance the performance of HRN. Our findings indicate a considerable improvement in results when relying on gold annotations. This observation underscores the potential for notable enhancement in multi-defendant LJP by improving the first-level reasoning.

To analyze the effect of the different levels of reasoning chains in HRN, we conduct an ablation study. Table 3 shows the results on MultiLJP with five settings: (i) w/o CR: HRN without predicting criminal relationships. (ii) w/o SC: HRN without predicting sentencing circumstances. (iii) w/o FP: HRN without the forward prediction process and predicts law articles, charges, and terms of penalty in reverse order. (iv) w/o BV: HRN without the backward verification process and predicts law articles, charges, and terms of penalty in order. (v) w/o all: HRN degrades to mT5 with multi-task settings by removing all reasoning chains.

Table 3 shows that all levels of reasoning chains help HRN as removing any of them decreases performance:

• •

  Removing the first-level reasoning chain. We observe that both criminal relationships and sentencing circumstances decrease the performance of HRN when we remove the first-level reasoning chains. Specifically, removing criminal relationships (CR) negatively impacts performance, especially on law articles and charges prediction, which means criminal relationships are helpful for distinguishing law articles and charges; removing sentencing circumstances (SC) negatively impacts performance, especially on terms of penalty, which means sentencing circumstances are helpful for distinguishing terms of penalty.

• •

  Removing the second-level reasoning chain. We observe that the model without the second-level forward prediction process (FP) or the second-level backward verification process (BV) faces a huge performance degradation in multi-defendant LJP. As a result, although the model still performs the first-level reasoning, the absence of modeling forward or backward dependencies between LJP subtasks leads to poor LJP performances.

• •

  Removing all reasoning chains. When removing all reasoning chains from HRN, there is a substantial drop in the performances of multi-defendant LJP. Experimental results prove that hierarchical reasoning chains can be critical for multi-defendant LJP.

We also conduct a case study to show how multi-defendant reasoning chains help the model distinguish judgment results of different defendants and make correct predictions. Figure 3 shows prediction results for two defendants, where red and green represent incorrect and correct predictions, respectively. Defendant B helped A beat the victim, but the fact description does not show a direct crime against the victim. Without determining criminal relationships and sentencing circumstances, MAMD focuses on the tailing behavior around A and misclassifies A’s law articles, charges, and terms of penalty as article 264, theft, and 11 months. In contrast, by following reasoning chains to determine criminal relationships, sentencing circumstances, law articles, charges, and terms of penalty, HRN distinguishes different judgment results between defendants.