Lex Rosetta: Transfer of Predictive Models Across Languages, Jurisdictions, and Legal Domains

This paper explores the ability of multi-lingual sentence embeddings to enable training of predictive models that generalize beyond individual languages, legal systems, jurisdictions, and domains (i.e., contexts). We propose a new type schema for functional segmentation of adjudicatory decisions (i.e., decisions of trial and appellate court judges, arbitrators, administrative judges and boards) and use it to annotate legal cases across eight different contexts (7 countries, 6 languages). We release the newly created dataset (807 documents with 89,661 annotated sentences) including the annotation schema to the public.¹¹1https://github.com/lexrosetta/caselaw_functional_segmentation_multilingual

In the area of AI & Law, research typically focuses on a single context, such as decisions of a specific court on a specific issue within a specific time range. This is justified by the complexity of legal work and the need for nuanced solutions to particular problems. At the same time, this narrow focus can limit the applicability of the research outcomes, since a proposed solution might not be readily transferable to a different context. In text classification, for example, a model might simply memorize a particular vocabulary characteristic of a given context, rather than acquiring the semantics of a predicted type. Adaptation of such a model to a new context would then require the assembly of a completely new dataset. This may be both time-consuming and expensive, since the annotation of legal documents relies on legal expertise.

Certain tasks appear to be of interest to researchers from multiple countries with different legal traditions (e.g., deontic classification of legal norms embodied in statutory law, argument extraction from case law, summarization/simplification of legal documents, etc.). This suggests that there may be several core tasks in AI & Law that are of general interest in almost any context. One such task is a functional segmentation of adjudicatory decisions, which has been the subject of numerous studies in the past (see Section 2). In this paper, we show that for this particular task it is possible to leverage linguistic resources created in multiple contexts.

This has wide-reaching implications for AI & Law research. Since annotation of training data is expensive, models that are able to use existing data from other contexts might be instrumental in enabling real-world applications that can be applied across contexts. Such approaches may further enable international collaboration of researchers, each annotating their own part of a dataset to contribute to a common pool (as we do in this work) that could be used to train strong models able to generalize across contexts.

Segmenting court decisions into smaller elements according to their function or role is an important task in legal text processing. Prior research utilizing supervised machine learning (ML) approaches or expert crafted rules can roughly be distinguished into two categories. First, the task could be to segment the text into a small number of contiguous parts typically comprising multiple paragraphs (this work). Different variations of this task were applied to several legal domains from countries, such as Canada (Farzindar and Lapalme, 2004), the Czech Republic (Harašta et al., 2019), France (Boniol et al., 2020), or the U.S. (Savelka and Ashley, 2018). Second, the task could instead be labeling smaller textual units, often sentences, according to some predefined type system (e.g., rhetorical roles, such as evidence, reasoning, conclusion). Examples from several domains and countries include administrative decisions from the U.S. (Walker et al., 2019; Zhong et al., 2019), multi-domain court decisions from India (Bhattacharya et al., 2019), international arbitration decisions (Branting et al., 2019), or even multi-{domain,country} adjudicatory decisions in English (Šavelka et al., 2020). Identifying a section that states an outcome of the case has also received considerable attention separately (Xu et al., 2020; Petrova et al., 2020). To the best of our knowledge, existing work on functional segmentation of court decisions is limited to a single language—ours being the first paper exploring the task jointly on legal documents in multiple languages.

In NLP, the success of word embeddings was followed by an increasing interest in learning continuous vector representations of longer linguistic units, such as sentences (a trend that has been reflected in AI & Law research as well (Zhong et al., 2019; Westermann et al., 2021)). Multi-lingual representations recently attracted ample attention. While most of the earlier work was limited to a few close languages or pairwise joint embeddings for English and one foreign language, several approaches to obtain general-purpose massively multi-lingual sentence representations were proposed (Artetxe and Schwenk, 2019; Devlin et al., 2019; Conneau et al., 2020). Such representations were utilized in many downstream applications, such as document classification (Lai et al., 2019), machine translation (Aharoni et al., 2019), question answering (Lewis et al., 2020), hate speech detection (Aluru et al., 2020), or information retrieval (IR) in the legal domain (Zhebel et al., 2020). Our work is one of the first such applications in the legal domain and to the best of our knowledge the first dealing with more than two languages.

Approaches other than language-agnostic sentence embeddings (this work) were used in AI & Law research focused on texts in multiple languages. A recent line of work mapped recitals to articles in EU directives and normative provisions in Member states’ legislation (Nanda et al., 2020). There, mono-lingual models were used (i.e., one model per language). Other published applications in multi-lingual legal IR were based on thesauri (Dini et al., 2005; Sheridan et al., 1997). A common technique to bridge the language gap was the use of ontologies and knowledge graphs (Boella et al., 2015; Agnoloni et al., 2007; González-Conejero et al., 2018; Ajani et al., 2016). The multi-lingual environments, such as EU or systems established by international treaties, attracted work on machine translation (Koehn et al., 2009), meaning equivalence verification (Tang and Kageura, 2019), and building of parallel corpora (Steinberger et al., 2014; Sugisaki et al., 2016).

In creating the dataset, the first goal was to identify a task that would be useful across different contexts. After extensive literature review, we identified the task of functional segmentation of adjudicatory decisions as a viable candidate. To make the task generalizable, we decided to include only a small number of core types.

1. (1)

   Out of Scope – Parts outside of the main document body (e.g., metadata, editorial content, dissents, end notes, appendices).

2. (2)

   Heading – Typically an incomplete sentence or marker starting a section (e.g., “Discussion,” “Analysis,” “II.”).

3. (3)

   Background – The part where the court describes procedural history, relevant facts, or the parties’ claims.

4. (4)

   Analysis – The section containing reasoning of the court, issues, and application of law to the facts of the case.

5. (5)

   Introductory Summary – A brief summary of the case at the beginning of the decision.

6. (6)

   Outcome – A few sentences stating how the case was decided (i.e, the overall outcome of the case).

We created detailed annotation guidelines defining the individual types as well as describing the annotation workflow (tooling, steps taken during annotation). Eight teams of researchers from six different countries (14 persons) were trained in the annotation process through online meetings. After this, each annotator conducted a dry-run annotation on 10 cases and received detailed feedback. Then, each team was tasked with assembling approximately 100 adjudicatory decisions. Each team developed specifications for the decisions to be included in their part of the dataset.

Four of the contexts were double-annotated by two annotators (Canada, Czech R., France, U.S.A. I); the remaining four by just one. Each team had at least one member with a completed law degree. When a team had more than one member, law students were allowed to be included.

A high-level description of the resulting dataset is provided in Table 1. It consists of eight contexts from seven different countries (two parts are from the U.S.) with 807 documents in six languages (three parts are in English). Most of the contexts include judicial decisions, while U.S.A. II was the only context that consisted solely of administrative decisions.There are considerable variations in the length of the documents. While an average document in the U.S.A. I context comprises of 530.6 sentences, an average document in the France context is about ten times shorter (59.0 sentences).

The four double-annotated parts enabled us to examine the inter-annotator agreement. Table 2 shows the raw agreement on a character level. While it appears that recognizing the Outcome was rather straightforward in the France and U.S.A. I contexts, it was more complicated in case of Canada and the Czech R. This might be due to a presence/absence of some structural clue. We also observe that in the Czech R. context it was presumably much easier to distinguish between the Background and Analysis than in case of the other three contexts.

In this paper, we focus on prediction of the Background, Analysis, and Outcome types. We decided to exclude the Introductory Summary type, since it is mainly present in the data from the United States. For the double-annotated datasets, we picked the annotations that appeared to be of higher quality (either by consensus between the annotators themselves or by a decision of a third unbiased expert).

We first removed all the spans of text annotated with either of the Out of Scope or Heading types. The removal of Out of Scope leaves the main body of a decision stripped of potential metadata or editorial content at the beginning of the document as well as dissents, or end notes at the end. The removal of the text spans annotated with the Heading type might appear counter-intuitive since headings often provide a clear clue as to the content of the following section (e.g., “Outcome”, “Analysis” etc.). We remove these (potentially valuable) headings because we want to focus on the more interesting task of recognizing the sections purely by the semantics of their constitutive sentences. This task is more challenging, and more closely emulates generalization to domains where headings are not used or not present in all cases, or are not reliable indicators.

The transformed documents are separated into several segments based on the annotations of the three remaining types. Each segment is then split into sentences.²²2We used the processing pipeline from https://spacy.io/ (large models). For the Czech language we used https://github.com/TakeLab/spacy-udpipe with the Czech model (PDT) from https://universaldependencies.org/. The output was further processed with several regular expressions. A different method was used for the French dataset, which consists of a few very long sections, internally separated by a semicolon. After consultation with an expert we decided to split the cases by the semicolon as well. A resulting document is a sequence of sentences labeled with one of the Background, Analysis, or Outcome types. The highlighted (green) part of Table 1 provides basic descriptive statistics of the resulting dataset per the individual contexts. Our final dataset for analysis consists of 807 cases split into 74,539 annotated sentences.

In our experiments we use the Language-Agnostic Sentence Representations (LASER) model (Artetxe and Schwenk, 2019) to encode sentences from different languages into a shared semantic space. Each document becomes a series of vectors which represent the semantic content of a single sentence. We use these vectors to train a bidirectional Gated Recurrent Unit (GRU) model (Cho et al., 2014) for predicting sentence labels.

The LASER model is a language-agnostic bidirectional LSTM encoder coupled with an auxiliary decoder and trained on parallel corpora. The sentence embeddings are obtained by applying a max-pooling operation over the output of the encoder. The resulting sentence representations (after concatenating both directions) are 1024-dimensional. The released trained model,³³3https://github.com/facebookresearch/LASER which we use in this work, supports 93 languages (including the six in our dataset) belonging to 30 different families and written in 28 different scripts. The model was trained on 223 million parallel sentences. The joint encoder itself has no information on the language or writing script of the tokenized text, while the tokenizer is language specific. It is even possible to mix multiple languages in one sentence. The focus of the LASER model is to produce vector representations of sentences that are general with respect to two dimensions: the input language and the NLP task (Artetxe and Schwenk, 2019). An interesting property of such universal multi-lingual sentence embeddings is the increased focus on the sentence semantics, as the syntax or other surface properties are unlikely to be shared among languages.

The GRU neural network (Cho et al., 2014) is an architecture based on a recurrent neural network (RNN) that is able to learn the mapping from a sequence of an arbitrary length to another sequence. GRUs are able to either score a pair of sequences or to generate a target sequence given a source sequence (this work). In a bidirectional GRU, two separate sequences are considered (one from right to left and the other from left to right). Traditional RNNs work well for shorter sequences but cannot be successfully applied to long sequences due to the well-known problem of vanishing gradients. Long Short-Term Memory (LSTM) networks (Hochreiter and Schmidhuber, 1997) have been used as an effective solution to this problem (the forget gate, along with the additive property of the cell state gradients). GRUs have been proposed as an alternative to LSTMs with a reduced number of parameters. In GRUs there is no explicit memory unit, and the forget gate and the update gate are combined. The performance of GRUs was shown to be superior to that of LSTMs in the scenario of long texts and small datasets (Yang et al., 2020), which is the situation in this work. For these reasons, we chose to use GRUs over LSTMs.

The overall structure of the employed model is shown in Figure 1.⁴⁴4The model was implemented using the Keras framework (https://keras.io/). Each case is transformed into a $1080\times 1024$ matrix. The number 1080 represents the maximum length (in sentences) of any case in our dataset. Shorter cases are padded to be of uniform length. The vectors are passed to the model in batches of size 32. They first go through a masking layer, which masks the sentences used for padding. The data is then passed to a bidirectional GRU model with 256 units and a dropout of $0.2$. Finally, the model contains a time-distributed dense output layer with softmax activation, which outputs the predicted label (i.e., Background, Analysis, Outcome) for each sentence. As a loss function, we use categorical cross-entropy. As optimizer we use the Adam algorithm (Kingma and Ba, 2014) with initial learning rate set to $4e^{-3}$ (reduced to $4e^{-4}$ once validation loss has stopped decreasing, with a patience of 50 epochs). We train the model for up to 1000 epochs. We halt the training once the validation accuracy has not increased in 80 epochs. For prediction, we use the best model as determined by validation accuracy.

We performed three experiments to test the hypotheses (Section 1.2). The first experiment (H1) focused on model generalization across different contexts (Out-Context). The second experiment (H2) assessed model robustness when trained on multiple pooled contexts different from the one where the model was applied (Pooled Out-Context). Finally, the third experiment (H3) analyzed the effects of pooling the target context data with data from other contexts (Pooled with In-Context). The different pools of training data, and the baselines we compare them against, are summarized in Table 3 and described in Sections 5.1-5.3.

Since our dataset is limited, we performed a 10-fold cross-validation. The folds were kept consistent across all the experiments. The experiments were conducted in the following manner:

1. (1)

   Index $i=1$ is set.

2. (2)

   Pool of training context(s) is selected (see Sections 5.1-5.3).

3. (3)

   A single test context is selected (see Sections 5.1-5.3).

4. (4)

   Eight of the folds from the training context(s) are used as training data (index different from $i$ and $(i+1)\bmod 10$).

5. (5)

   The folds with index $(i+1)\bmod 10$ from the training context(s) are designated as validation data.

6. (6)

   The $i$ fold from the test context is designated as test data.

7. (7)

   The models are trained and evaluated.

8. (8)

   Index $i=i+1$ is set.

9. (9)

   If $i\leq 10$ go to (4) else finish.

To evaluate the performance of the systems trained on data from different contexts, we used Precision ($P$), Recall ($R$), and $F_{1}$-measure.

For statistical evaluation we used the Wilcoxon signed-rank test (Wilcoxon, 1992) to perform pair comparisons between methods and the baselines as suggested by (Demšar, 2006). We used the overall (micro) $F_{1}$-score as the evaluation metric. The null-hypothesis states that the mean performance of the two compared methods (i.e., an assessed model and a baseline) is equivalent.

Since the number of samples is rather small (7 contexts in testing H1, and 8 contexts in testing H2 and H3), we are not able to reject the null hypotheses at $\alpha=0.05$ for any of H1–H3. This is as expected given the size of the dataset, and it is not unacceptable given the exploratory nature of our work at this stage. We will be in a position to formally evaluate the hypotheses once we extend the dataset (see Section 10). Instead, we report raw p-values produced by the testing as an evidence of the assessed methods’ effectiveness. The p-value is the probability of seeing the results at least as extreme as those we observed given the null hypothesis is correct (i.e., their mean performance is the same).

The results of all three experiments are shown in Table 4. Each row reports a performance of a specific model across different contexts (columns). Each cell shows a bold number on the first row which correspond to the (micro) average F₁-score over the three predicted classes, with standard deviation across the 10 folds. The F₁-scores of the three classes are reported in the second row of each cell ordered as Background, Analysis, Outcome. A visual explanation of cell contents can be found in Figure 3.

It appears that the multilingual sentence embeddings generated by the LASER model excel at capturing the semantics of a sentence. A model trained on a single context could in theory capture the specific vocabulary used in that context. This would almost certainly lead to poor generalization across different contexts. However, the performance statistics we observed when transferring the Out-Context models to other contexts suggest that the sentence embeddings provide a representation of the sentences that enable the model to learn aspects of the meaning of a sentence, rather than mere surface linguistic features.

The results clearly point to certain relationships where contexts within the same or related languages appear to work well together, e.g., {Canada, U.S.A. I, and U.S.A. II} or {Czech R. and Poland}. This could indicate that the multi-lingual embeddings work better when the language is the same or similar. It could also point to similarities in legal traditions, e.g. the use of somewhat similar sentences to indicate a transition from one type of section to the next. Note that we removed headings, which means that explicit cues could not be relied on by the models we trained on such transformed documents. Finally, the cause could also be topical (domain) similarity of the contexts (e.g., both the U.S.A. contexts deal with employment law). Also note that the above are just possible explanations. We did not perform feature importance analysis on the models.

To gain insight into this phenomenon, we visualized the relationships among the contexts on a document level. We first calculated the average sentence embedding for each document. This yielded 1024-dimensional vectors for 807 documents representing their semantics. We arranged the resulting vectors in a matrix ($1024\times 807$) and performed a Principal Component Analysis (PCA) reducing the dimensionality of the document vectors to 2. This operation enabled a convenient visualization shown in Figure 4.

Overall, the cases from the same contexts appear to cluster together. This is expected as the documents written in the same language, having similar topics, or sharing some similarities due to legal traditions, are likely to map to vectors that are closer. The documents from both the U.S.A. contexts occupy the same region in the embedding space. This is not surprising as they come from the same jurisdiction, are written in the same language, and deal with similar topics (employment law). The Canadian cases, which are also in English, occupy a nearby space. This could be linked to the language as well as likely similarities in legal traditions. French, German, and Italian documents occupy the middle space; they are closer to the English documents than those from the Czech R. and Poland. Interestingly, the Czech and Polish documents occupy almost the same space. The Polish context focuses on the rule of law while the Czech one is supposed to be more general. As the latter deals with the decisions of the top-tier courts (one of them Constitutional), it is possible that the topics substantially overlap. Moreover, Poland and the Czech R. share similar legal traditions and languages from the same family (Slavic), so the close proximity of the documents in the embedding space might not be unexpected. Finally, German, French, and Canadian cases occupy wider areas than documents from other contexts. This could be due to their lack of focus on specific legal domains.

We observed a peculiar phenomenon where the Out-Context models trained on the German and Polish contexts failed to detect Outcome sentences on the six remaining contexts and vice-versa. The cause is readily apparent from the visualization shown in Figure 5. Each segment of the figure depicts a spatial distribution of sentences color coded with their labels across documents for a particular context. As can be seen, the cases typically follow a pattern of a contiguous Background segment, followed by a long Analysis section. The several Outcome sentences are placed at the very end of the documents. In the Polish and German decisions, however, the Outcome sentences come first. The GRU models we use rely on the structure as well as semantics in making their predictions. As we can see, a model trained exclusively on cases that begin with a Background might therefore have difficulties correctly identifying outcome sections at the beginning, and vice-versa. However, as we will see below, a model trained with data featuring both structures can learn to correctly identify the correct structure based on the semantics of the sentences.

The model trained on the French context appears to perform better on detecting Outcome sentences than models trained on other contexts. This is somewhat surprising as the French model’s overall performance is among the weakest (e.g. compare the Czech model’s average F${}_{1}=0.77$ to the F${}_{1}=0.68$ of the French model). Again, Figure 5 provides an insight into why this happens. The French context is the only one where the count of Outcome sentences is comparable to those of the other two categories. For all the other contexts, the Outcome sentences are heavily underrepresented. This reveals an interesting direction for future work where the use of re-sampling may yield models with better sensitivity for identifying Outcome sentences.

In two instances models trained on a single context under-performed the random baseline. The model trained on the Germany context achieved the average F${}_{1}=0.29$ when applied to the French context (Random $F_{1}=0.33$). As the model trained on Polish data also performed poorly on the French context ($F_{1}=0.38$) the cause appears to be the inability of the two models to detect the Outcome sentences at the end (discussed above). As the Outcome sentences are heavily present in the French context, this problem manifests in a score lower than the random baseline. The second instance is the model trained on the Italy context applied to the U.S.A. I data (F${}_{1}=0.53$ versus F${}_{1}=0.58$ on Random). Here, the cause appears to be different. Note that the Italian context likely has a very specific notion of Outcome sentences (F${}_{1}=0.94$ on Italy$\rightarrow$Italy). It appears that many Analysis sentences from the U.S.A. I context were labeled as Outcome by the model. Summary judgments often address multiple legal issues with their own conclusions which could have triggered the model to label such sentences as Outcome.

An important finding is the performance of the Pooled Out-Context model (H2) shown in the Pooled row of Table 4. The experiment simulates training of a model on several contexts, and then applying it to an unseen context. The Pooled Out-Context models, having no access to the data from a target context, reliably outperform the best single Out-Context models. They appear to be competitive with several In-Context models. These results are achieved with a fairly small dataset of 807 cases. We expect that expanding the dataset would lead to further improved performance.

The Pooled with In-Context experiment (H3) models the situation where data from a target context is available in addition to labeled data from other contexts. Our experiments indicate that the use of data from other contexts (if available) in addition to data from the target context is preferable to the use of the data from the target context only. This is evidenced by the improved performance of the models trained on the pooled contexts over the single In-Context models. The models have an interesting property of being able to identify the Outcome sentences with effectiveness comparable to (or higher than) the models trained on the same context only. This holds for all the contexts, except Poland where the Outcome performance is a bit lower (0.65 vs. 0.88). This indicates that the model is able to learn the two possible modes of the Outcome section placement. It successfully distinguishes cases where the section is at the beginning from the cases where the Outcome sentences are found toward the end.

The inclusion of the In-Context data in the pooled data leads to a remarkable improvement over only using the pooled Out-Context data. The magnitude of the improvement highlights the importance of including such data in the training. We envision that the models trained on different contexts used in combination with high-speed similarity annotation frameworks (Westermann et al., 2019, 2020) could enable highly cost efficient annotation in situations where resources are scarce. Perhaps, adapting a model to an entirely new context could be as simple as starting with a model trained on other contexts, and spending a few hours correcting the misconceptions of the model to teach it the particularities of the new context.

We analyzed the use of multi-lingual sentence embeddings in sequence labeling models to enable transfer across languages, jurisdictions, legal systems (common and civil law), and domains. We created a new type schema for functional segmentation of adjudicatory decisions and used it to annotate legal cases across eight different contexts. We found that models generalize beyond the contexts they were trained on and that training the models on multiple contexts increases their robustness and improves the overall performance when evaluating on previously unseen contexts. We also found that pooling the training data of a model with data from additional contexts enhances its performance on the target context. The results are promising in enabling re-use of annotated data across contexts and creating generalizable and robust models. We release the newly created dataset (807 documents with 89,661 annotated sentences), including the annotation schema and the code used in our experiments, to the public.

This work suggests a promising path for the future of international collaboration in the field of AI & Law. While previous annotation efforts have typically been limited to a single context, the experiments presented here suggest that researchers can work together by annotating cases from many different contexts at the same time. Such a combined effort could aid researchers in creating models that perform well on the data from the context they care about, while at the same time helping other groups train even better models for other contexts. We encourage these research directions and hope to form such collaborations under the Lex Rosetta project.

The application of multi-lingual sentence embeddings to functional segmentation of case law across different contexts yielded promising results. At the same time, the work is subject to limitations and leaves much room for improvement. Hence, we suggest several directions for future work:

• •

  Extension of the datasets from different contexts used in this work beyond ~100 documents per context.

• •

  Annotation of data from contexts beyond the eight used here (multi-lingual models support close to 100 languages).

• •

  Analysis of automatic detection of Introductory Summary, Headings, and Out of Scope.

• •

  Identification and investigation of other tasks applicable across different contexts.

• •

  Evaluation of the application of other multilingual models (e.g., those mentioned in Section 2).

• •

  Exploring other transfer learning strategies beyond simple data pooling, such as the framework proposed in (Šavelka and Ashley, 2015).

• •

  Using multi-lingual models for annotation tasks with high-speed annotation framework, such as (Westermann et al., 2019, 2020).

• •

  Performing the transfer across contexts with related (but different) tasks, such as in (Šavelka et al., 2020).

• •

  Further exploring the differences in the distribution of the multilingual embeddings for purposes of comparing and analyzing domains, languages, and legal traditions.