MILPaC: A Novel Benchmark for Evaluating Translation of Legal Text to Indian Languages

We develop this dataset from a set of primers released by a society of Law practitioners – the WBNUJS Intellectual Property and Technology Laws Society (IPTLS) – from a reputed Law school in India. The primers are released in English (EN) and 9 Indian languages, viz., Bengali (BN), Hindi (HI), Marathi (MR), Tamil (TA), Gujarati (GU), Telugu (TE), Malayalam (ML), Punjabi (PA), Oriya (OR). Each primer contains approximately 57 question-answer (QA) pairs related to Indian Intellectual Property laws, each QA pair given in all the Indian languages stated above.

The 10 PDF documents (one corresponding to the primer in each language) were downloaded from the website of the IPTLS society stated above.⁷⁷7https://nujsiplaw.wordpress.com/my-ip-a-series-of-ip-awareness-primers/ The PDF documents had embedded text which was extracted using the pdftotext⁸⁸8https://www.xpdfreader.com/pdftotext-man.html utility. Regular expressions were used to identify the start of each question and answer. Each QA pair in these primers was numbered consistently across all languages, and we used this numbering to identify parallel textual units.

Each question and each answer is considered as a ‘textual unit’ in this dataset. Note that, we decided to not break the questions/answers into sentences, due to the following reasons: First, we have alignments in terms of questions and answers, and not their individual sentences. If we split each question and answer into sentences, aligning the sentences would require synthetic methods (like nearest neighbour search) which might have introduced some inaccuracies in the dataset. Since the primers are a high-quality parallel corpus developed manually by Law practitioners, we decided not to use such synthetic methods to maintain a high standard of the parallel corpus. Second, since this dataset contains QA pairs, it can be used for other NLP tasks such as Cross-Lingual Question-Answering. Tokenizing the text units into sentences would remove this potential utility of the dataset.

The number of parallel pairs in the MILPaC-IP dataset for various language-pairs is given in Table 4 (black entries in the upper triangular part). The slight variations in the entries for different language-pairs in Table 4 are explained in Appendix A.1. Some examples of text pairs of the MILPaC-IP are given in Figure 2(a).

The second parallel corpus is developed from a set of FAQ booklets released by the Competition Commission of India (CCI). These FAQ booklets are based on The Competition Act, 2002 statute that provides the legal framework to deal with competition issues in India. The FAQs are written in English and 4 Indian languages – Bengali, Hindi, Marathi, and Tamil. Each FAQ booklet contains 184 QA pairs. Each FAQ booklet has been manually translated by CCI officials and can be treated as expert translation, which makes this a reliable source for authentic multilingual parallel data.

The FAQ booklets (PDF documents) corresponding to each language were downloaded from the CCI website⁹⁹9https://www.cci.gov.in/advocacy/publications/advocacy-booklets. These documents are not machine readable; hence, the text was extracted using OCR (as done by Ramesh et al. (2022)). We tried out two OCR systems - Tesseract¹⁰¹⁰10https://github.com/tesseract-ocr/tesseract and Google Vision API¹¹¹¹11https://cloud.google.com/vision/docs/samples/vision-document-text-tutorial. Both OCR systems support English and all 9 Indian languages considered in this work. An internal evaluation (details in Appendix, Section C) showed that the Google Vision had better text detection accuracy than Tesseract. Hence, we decided to use Google Vision.

After applying OCR to the PDF documents, a single text file is obtained, which is annotated to separate and align the text units. For all QA pairs, the translations had the same question number in all the documents, and this information was used to align the QA pairs. Note that, as in MILPaC-IP, we do not split the questions and answers into sentences for the same reasons as discussed above.

The number of parallel text pairs for various language-pairs in the MILPaC-CCI-FAQ dataset are given in the Table 4 (blue italicized entries in lower triangular part). As stated earlier, each FAQ booklet contains 184 QA pairs. The answers for 3 QA pairs are given in tables; for these, we did not include the answers and considered only the questions, because of difficulties in accurate OCR of tables. Hence, for each language-pair, we have $184\times 2-3=365$ text pairs, as stated in Table 4. Some example text pairs are given in Figure 2(b).

This dataset is made up from PDF scans (non-machine readable) of Indian Acts (statutes or written laws) documents published by the Indian Legislature. Unlike the previous two datasets (which are already compiled in multiple languages), this parallel corpus is not available readily, and is very challenging and time-consuming to develop.

The first challenge is that, while a lot of Indian Acts are available in English and Hindi, very few are available in other Indian languages. Even among the few Acts that are available in other Indian languages, only a small fraction are suitable for OCR (most other PDFs are of poor texture and resolution). The challenges in OCR of Indian Legal Acts are described in the Appendix (section B).

The English and translated versions of the Acts were obtained from the website of the Indian legislature¹²¹²12https://legislative.gov.in/central-acts-in-regional-language/. Candidate Acts were selected based on two criteria – (i) availability of translations in multiple languages (so as to get translation-pairs across Indian-Indian languages as well), and availability of good quality scans. 10 Acts were selected as per these criteria; the selected Acts are listed in Appendix, Section A.2.

As discussed previously for the MILPaC-CCI-FAQ dataset, Google Vision was chosen for OCR. A similar process was followed to get one text file corresponding to each Act, and the extracted text was then annotated using the same semi-automatic approach as was described for MILPaC-CCI-FAQ. We identified parallel text units (paragraphs, bulleted list elements) by looking at the versions in the two languages and using structural information such as paragraphs, and section/clause numbering. Table 5 gives the number of parallel text pairs for different language-pairs in MILPaC-Acts. Some example text pairs of MILPaC-Acts are given in Figure 2(c).

Note that, compared to MILPaC-IP and MILPaC-CCI-FAQ datasets, aligning the Acts documents is much more challenging and requires frequent manual intervention. The documents are long, and often have multi-column text layout, which confuses OCR systems leading to text segmentation issues (see Appendix, Section B for the main challenges in OCR). Such factors frequently lead to OCR errors that need to be manually corrected by rearranging the extracted text by referring back to the source document. Hence developing this dataset is very time-consuming. This is why, till date, we have been able to process only 10 Acts. Even after the manual corrections, a fraction of the data had to be discarded since they could not be aligned across languages. This non-alignment was mainly due to the factors that are explained in Appendix Section B. According to our estimate, in the MILPaC-Acts, approximately 10% of the textual units had to be discarded on average across all the Acts we considered. The percentage of text units discarded for individual Acts varies between 8% and 14%. All these steps have been taken to ensure the high quality of the dataset, making it very time-consuming to develop.

We have attempted to make MILPaC a high-quality parallel corpus for evaluating MT systems, through the following steps.

The MILPaC corpus is publicly available at https://github.com/Law-AI/MILPaC under a standard non-commercial license. All the legal documents and translations used to develop the MILPaC datasets are publicly available data on the Web. We have obtained formal permissions for using and publicly sharing the datasets – from the Competition Commission of India (CCI) for the MILPaC-CCI-FAQ dataset, from the WBNUJS Intellectual Property and Technology Laws Society (IPTLS) for the MILPaC-IP dataset, and from the Ministry of Law and Justice, Government of India for using and publicly sharing the MILPaC-Acts dataset.

We consider the following MT systems, some of which are commercial, while others are publicly available MT systems. These systems were selected because of their strong performances on well-known MT benchmarks such as FLORES-101 (Goyal et al., 2021), WAT2021¹⁵¹⁵15https://lotus.kuee.kyoto-u.ac.jp/WAT/WAT2021/index.html, and WAT2020 (Nakazawa et al., 2020).

There are 2 variants available for IndicTrans – Indic-to-English and English-to-Indic which support 11 Indian languages, including all 9 of the Indian languages present in our corpus. The English-to-Indic model is used for our experiments. We refer the readers to the original paper (Ramesh et al., 2022) for more details.

We use the following standard metrics to evaluate the performance of MT systems:

We scale all the three metric values to the range $[0,100]$, to make them more comparable.

Table 8 shows the performances of all the MT systems across the 3 datasets of MILPaC for the languages Bengali (BN), Hindi (HI), Tamil (TA), and Marathi (MR). Table 9 shows the performances of all the MT systems over MILPaC-IP and MILPaC-Acts for the languages Telugu (TE), Malayalam (ML), Punjabi (PA), Oriya (OR), and Gujarati (GU) (since MILPaC-CCI-FAQ does not have these languages).²⁸²⁸28All experiments were conducted on a machine having 2 x NVIDIA Tesla P100 16GB GPU.

Note that the Davinci-003 and GPT-3.5T-Inst have a maximum bound on the number of tokens (as stated earlier). Hence, out of the 1,460 EN-IN pairs in the MILPaC-CCI-FAQ dataset, the 77 longest text units could not be fed into the Davinci-003 and GPT-3.5T-Inst models in their entirety (these include 19 text units in BN, 15 in HI, 31 in TA, and 12 in MR). One option was to break these long text units into chunks and then translate chunk-wise, but we preferred not to translate chunk-wise in order to get a fair comparison with other models. Hence, the 77 longest text units were not considered while evaluating Davinci-003 and GPT-3.5T-Inst models over the MILPaC-CCI-FAQ dataset. For all other MT systems, the values reported are averaged over all text-pairs in each dataset.

We find that no single model performs the best in all scenarios. MSFT, GOOG, and IndicTrans are the 3 best models that generally perform the best in most scenarios. It can be noted that the same trend was observed by (Ramesh et al., 2022) for Indian non-legal text, i.e., IndicTrans, GOOG, and MSFT outperform mBART-50 and OPUS. A notable exception is that NLLB performs the best for EN-OR translation, possibly because Oriya is a resource-poor language that NLLB is specially designed to handle.

The translation performances of both Davinci-003 and GPT-3.5T-Inst are relatively poor. One particular problem we observed in the translations using these two LLMs is that, for longer text units, both models sometimes repeat multiple times the translation of the last few words in their translated text.

The scores for MILPaC-Acts are consistently lower than those for other datasets. This is expected since MILPaC-Acts has very formal legal language, which is challenging for all MT systems. Interestingly, though MSFT and GOOG perform the best over most datasets, IndicTrans performs better over MILPaC-Acts for several Indian languages (e.g., Malayalam & Gujarati). The superior performance of IndicTrans over MILPaC-Acts may stem partly from the fact that it was trained on some legal documents from Indian government websites (such as State Assembly discussions) according to (Ramesh et al., 2022). However, it is not known publicly over what data commercial systems such as GOOG and MSFT are trained.

Apart from the automatic evaluation of translation performance (as detailed above), we also evaluate the performance of MT systems through a set of Law students. This is an important part of the evaluation, since we can get an idea of how well automatic translation performance metrics agree with the opinion of domain experts.

The Law students who participated in our human surveys were clearly informed of the purpose for which the annotations/surveys were being carried out. They consented to participation in the survey and agreed that automatic translation tools can be of great help in the Indian legal scenario. Each of them was paid a mutually agreed honorarium for the evaluation. We took all possible steps to ensure that their annotations are unbiased (e.g., by anonymizing the names of the MT systems whose outputs they were annotating) and accurate (e.g., by asking them to write justifications for low scores, post-survey discussions, etc.). We also conducted a pilot survey (described below) and discussed with them after their evaluation to get their feedback. Through all these steps, we attempted to ensure that the annotations are done rigorously.

The Law student who is a native speaker of Bengali, was not available during the evaluation of MILPaC-Acts. Thus, in total, we performed human evaluations for 1,050 text pairs [(50 text units of MILPaC-CCI-FAQ x 3 MT systems x 4 language pairs) + (50 text units of MILPaC-Acts x 3 MT systems x 3 language pairs)].

However, we understood from our discussions with the Law students that, for evaluating translations in the Legal domain, a third metric (SLU: Suitability for Legal Use) is also essential. A translation may perfectly capture the meaning of the source text, but if the language of the translation is not keeping with the Legal standards, then the translated text cannot be used for purposes such as legal drafting. This difference arises mainly because legal language needs to be very formal and unambiguous, particularly while drafting laws and guidelines. Sometimes exact words and phrases need to be used, whereas if a translation uses a synonym or a similar phrase, then the translation will not be considered suitable for legal use, even if the meaning of the source text is well captured. This is why, after discussion with the Law students, we decided to use two separate metrics – Preservation of Meaning (POM) and Suitability of Legal Use (SLU) – during our human survey.

It can be seen from Table 10 that for the MILPaC-Acts dataset, SLU scores are substantially lower than the POM scores, for all Indian languages and all MT systems. This means that even when the translations well captured the meanings of the source text, the translated texts are not sufficiently formal for legal drafting, as per the opinion of the Law students. This problem is particularly important for the MILPaC-Acts dataset which contains very formal legal text that the MT systems often fail to generate. On the other hand, the SLU scores are much closer to the POM scores for the MILPaC-CCI-FAQ dataset, possibly because this dataset contains simplified versions of legal text. It is an interesting direction of future research as to how the legal translations can be made sufficiently formal when necessary, e.g., when the formal laws of a country are to be translated.

It is to be noted that different Law practitioners have different preferences about translations of legal text (as shown by the moderate IAA between the two annotators for English-to-Hindi translations). Hence the observations presented above are according to the Law students we consulted, and can vary in case of other Law practitioners.

The correlation values corresponding to each language pair are given in Table 11 for MILPaC-CCI-FAQ and MILPaC-Acts datasets respectively.²⁹²⁹29As stated earlier, the Law student who is a native speaker of Bengali, was not available during the expert evaluation of MILPaC-Acts. We observe the correlations of automatic metrics and human scores to be moderate for MILPaC-Acts (between $0.35$ and $0.4$ on average). Only for EN-HI, correlations are higher (above $0.6$). Whereas, for MILPaC-CCI-FAQ, all correlations were very low, around $0.1$ on average. Among the automatic metrics, chrF++ shows slightly higher correlation with human scores than GLEU. Such low correlations with human scores indicate the need for improvement in the automatic evaluation of MT systems in the legal domain.

As part of our survey (that has been described earlier), we also asked the Law students to give, along with the POM, SLU, and FLY scores, comments to explain why they gave certain translations low scores. We also asked them to identify different types of errors committed by the MT systems. They identified various types of errors such as inclusion of extra words in translations, mis-translation of terms, presence of untranslated portions, consistency errors (inconsistency between the factual information in the original and translated text), and so on. Table 12 shows the common errors and the percentage of text-pairs in which each error was identified by the Law students (across text-pairs in all Indian languages, that were evaluated by them). GOOG and MSFT frequently have untranslated portions (which possibly led to their poor expert scores). Most errors by IndicTrans were minor punctuation errors, which did not affect the translation quality much; however IndicTrans often mis-translates legal terms. We hope that these observations will help in improving these MT systems in future.

Some examples of errors committed by the MT systems have been shown in Figure 3. For each example, the POM, SLU, and FLY scores, along with the comments written by the Law students (who gave the scores), explaining the problem(s) / error(s) in the translated text, are reported. We see examples where the Law students opined that the meaning of the original text has not been captured in the translation, examples where a portion of the original text has not been translated, and examples where certain terms having domain-specific significance have not been properly translated.

Translation of legal text from one Indian language to another may be necessary, e.g., in scenarios where a person who is a native of one Indian state (and understands the local language of his/her native state) becomes a party to a legal case in a lower court of another state having a different local language. MILPaC can be used for evaluating Indian-to-Indian language translation in the legal domain as well. To demonstrate this, we investigate the question – while translating between two low-resource Indian languages, is it better to translate directly, or through a high-resource intermediary language?. While answering this question systematically will require a full-fledged study, here we briefly investigate the question using only the MILPaC-IP dataset and the GOOG and MSFT systems.

We select a low-resource Indian language-pair (e.g., Punjabi to Marathi translation), and then consider two approaches for this translation – (1) direct translation (PA $\rightarrow$ MR) and (2) translation through English as an intermediary language (PA $\rightarrow$ EN $\rightarrow$ MR). Table 13 compares these two approaches for three Indian language-pairs – Punjabi to Marathi translation, Gujarati to Marathi translation, and Bengali to Telugu translation. Interestingly, for GOOG, direct translation is consistently better, but for MSFT, going through English marginally improves performance. While more experimentation is needed for a detailed understanding, we here demonstrate that MILPaC can be used to address such questions on legal translation from one Indian language to another.

The datasets in MILPaC are much smaller in size compared to general-domain parallel corpora, due to two primary reasons. First, developing such corpora requires the involvement of Law practitioners which is expensive and limited. Second, creating these datasets required a lot of manual effort, due to the poor quality of the documents (specially in the MILPaC-Acts dataset) and the consequent challenges in OCR of these documents (detailed in the Appendix). Thus, the MILPaC datasets are not suitable for training Neural MT systems from scratch. Hence we are proposing this corpus mainly for the evaluation of MT systems. However, in this section, we show that MILPaC is also suitable for fine-tuning MT systems to improve their performance over Indian Legal text.

To this end, we randomly split MILPaC (combining all 3 datasets) in a 70 (train) - 10 (validation) - 20 (test) ratio language-wise. For instance, we combined the English-Bengali text-pairs from all 3 datasets, and then split this combined set to get the train-test-validation splits for the Bengali language. Then, for each Indian language, we fine-tuned the open-source MT model IndicTrans (Ramesh et al., 2022) with the 70% train set (combined 70% of all 3 MILPaC datasets). We used the 10% validation split (combined 10% of all 3 datasets) to select the best checkpoint among different iterations and to prevent overfitting by early stopping with the patience value as 3. Then we checked the performance of the fine-tuned IndicTrans (which we call ‘IndicTrans-FT’) over the 20% test set (20% of the 3 datasets combined).

Table 14 compares the performance of the off-the-shelf IndicTrans (Ramesh et al., 2022) and the fine-tuned IndicTrans-FT over the 20% test set for 4 Indian languages. For every language, we find that IndicTrans-FT gives substantial improvements over the off-the-shelf IndicTrans. Thus, though the intended primary use of MILPaC is to evaluate MT systems, these results show that MILPaC can also be used to fine-tune general domain MT systems to improve their translations in the legal domain.