Towards Automatic Comparison of Data Privacy Documents: A Preliminary Experiment on GDPR-like Laws

The General Data Protection Regulation (GDPR) came into effect in 2018 to replace the Data Protection Directive. It was designed to leverage data privacy laws across Europe with the development of new technology and the new challenges and problems in protecting personal privacy (EUd, 2018). The structure of GDPR consists of Chapters, Articles, Sections and Recitals. It applies to all the organizations that handle personal data for EU residents, regardless of their locations and thus, it is widely viewed as a benchmark for data protection and privacy regulations and inspired many countries to legislate their own laws to enable greater protection and capabilities for users. For example, Brazil passed the General Data Protection Law (Portuguese: Lei Geral de Proteção de Dados Pessoais, or LGPD) in 2018, and it came into effect in February 2020. India also release their law for user protection, Personal Data Protection Bill (PDPB), which is modeled after the GDPR, though there are significant differences between the Indian and European law.

To perform a sentence similarity analysis, information retrieval is a key to success since legal documents are generally long. Text summarization is a potential direction for this task since it selects important sentences to represent the entire document. However, summarizing the long document is still a very difficult task. Some research studies (Cachola et al., 2020; Cagliero and La Quatra, 2020) have proposed a transformer-based and n-gram-based model to summarize long scientific documents but this is limited to use for only scientific papers and incapable of legal document summarization.

Computing the similarity between two legal documents is a very important task in the domain of legal document processing such as information retrieval and legal documents comparisons. This task also has a lot of important downstream applications (e.g. recommendation and ranking). Previous models of legal document similarity mainly focus on two directions. First, regardless of text content, a similarity score can be measured by analyzing the dominant citations network. Kumar et al. (2011) proposes a model that is extended to the popular techniques used in information retrieval and search engines using co-citation. Second direction is comparing the content of legal documents using text similarity algorithms. In short, we feed a pair of documents and the algorithm outputs the similarity score indicating how similar they are. A number of researchers exploit the task as an instance of the semantic text matching (STM) problem. Landthaler et al. (2018) proposes a purely unsupervised solution for semantic text matching between contract clauses and legal comments. Mandal et al. (2017) further develops a model that takes advantage of TF-IDF-based methods and advanced techniques such as word embedding and document embedding. However, the content similarity is still a hard task as understanding the content itself is already challenging.

Many works of literature study case-based instead of document-based where the task is to extract similar or relevant cases “S1, S2, S3, …, Sn” from historical legal documents given the new case “Q” (Tran et al., 2019; Nguyen et al., 2020). In addition to case-based research, a number of studies focus on statute law documents where the task is to extract relevant articles given the question “Q” (Dang et al., 2019; Gain et al., 2021; Nguyen et al., 2020) and the following task is to answer a given question “Q” by determining if the relevant articles entail ”Q” or ”not Q” (Hudzina et al., 2019; Gain et al., 2021; Nguyen et al., 2020). These works are closely related to our task but there are some minor differences including (i) we compare two documents instead of having an input “Q” then find relevant articles from a corpus of legal documents, (ii) the literature mainly focuses on cases or statute laws while we focus on the new type of law and regulation for digital privacy, and (iii) many-to-many comparison is a much harder problem than one-to-many because we also need to rank the similar articles of document “D1” for every article in the document “D2”. This means a ranking algorithm (Narayan et al., 2018) might be required for this task.

In order to fully exploit the text and better compare similarities, we use BERT (Devlin et al., 2019), a pre-trained transformer network, which set for various NLP tasks new state-of-the-art results, including question answering, sentence classification, and sentence-pair regression. Specifically, we convert each sentence into embedding space and then use metrics such as cosine similarity for similarity comparison. Different than previous models which use only word-based approaches, our algorithm is expected to be able to better analyze text and thus gain a better similarity. We will investigate more potential directions along the way. To simplify our study, we focus on English legal documents as the first step. We put the extension to other languages as our future work.

We extract content in natural language and form a well-structured data. The GDPR is comprised of four levels of content. Chapter is the top hierarchy of the law structure, each chapter has many sections. Each section has many articles and there are many recitals in each articles. Note that each recital can include multiple sentences. We extract content into recital-level as it is the smallest piece of information to convey some legal meaning. Because some recitals themselves have no meaning without the relevant contexts, performing document analysis on the upper level (e.g. articles level) would be more informative.

In this section, we propose several algorithms from traditional ones to deep learning ones for document comparison. We choose TF-IDF, which is widely used and still one of the most popular ones for legal documents comparison. Then we will introduce deep learning based algorithms.

We choose the Brazilian GDPR-like law (LGPD) to compare to the original GDPR (European version). There are two reasons for this. First, the originally, LGPD was written in Spanish then translated into English version. This helps us by short-cutting the translation step from non-English to English language in order to compare with GDPR. Second, we investigated LGPD and found that it has the same document structure as GDPR including Chapter, Section, Article and Recital. The example of LGPD is shown in Figure 1.

To evaluate system in the document similarity ranking, we use HIT@K evaluation method. It would determine the correct match only if among the top K similar document retrieved from corpus, there is at least one document is actually the right match. Formally, given an input document $d^{a}$ from corpus $C_{a}$, we get $K$ most similar documents $d^{b}_{1}$, $d^{b}_{2}$, …, $d^{b}_{K}$ retrieved from the other corpus $C_{b}$. Equation 4 shows the accuracy as average of corrects where $n$ is the total number of documents in $C_{a}$ that we evaluate. The prediction is determined as correct if at least one of top K similar documents $d^{b}_{j}$ where $j\in[1,K]$ is matched to the input $d^{a}$.

In this paper, we choose HIT@1 (we set $K=1$) as our evaluation metric. We conduct comparison experiment in article-level and recital-level because we think that directly using similarity algorithms would not be able to handle very long documents. The example of correct match and incorrect match in article-level are shown in Figure 2.

There are over 80 articles in GDPR, so we choose to evaluate 10 articles as our pilot test. These articles are from different topics including Subject matters, Material scope, Territorial scope, Definition of personal data, Principles of personal data usage, consent, Conditions for consent, Child’s consent, Special types of personal data and Personal data relating to criminal. Two researchers manually check all these articles and their matches in order to determine the corrects. If we have disagreement, we discuss and finalize the agreed labels. The accuracy is the average corrects of these ten manually labeled article pairs.

We report our preliminary results based on experimental setup and evaluation methods described in Section 4.1 and  4.2, respectively. Table 1 shows the result of accuracy for GDPR and Brazilian GDPR (LGPD). Both experimental results on recital and article level are included.

For recital level, the result from the models could vary for different topics since we are using only one or a few sentences. It could be affected by the noise of texts, the length of the recitals, etc. But it still depicts the effects of different models. BERT gives the best result in terms of accuracy. This matches the recent NLP papers that pretrained model on big data using deep learning technique can provide a better sense and representation of texts. Word embedding performs the worst. We think this is because 1) GloVe, the model we use for generating embedding space is relatively old 2) word embedding is trained on general texts such as wikipedia, etc, it probably can not improve legal documents. However, more sentences can better reflect the model. We will analyze this in the next section. The result for TF-IDF is similar to other papers where models often rely on this feature to find related or similar documents. Siamese embedding performs the second best, as TF-IDF, although the result is supposed to be better since this model is trained on gold label data. The reason is because that Siamese BERT is trained on pairs of sentences. For these sentences, words such as “should” and “must” will have a very similar representation. This makes general texts more understandable but for law documents, this can get inaccurate.

For article level, the effect, no matter a good effect or a bad one, from the models will be accumulated and thus the results will reflect the algorithm quality better. BERT gives the best result. This matches the recital level and further confirm that recent NLP developments with deep learning technique can help the legal document tasks. Word embedding and TF-IDF are similar. We think that since word embedding is trained on general texts, it should just be a reflection of the general word, just like TF-IDF using word count and similar techniques. Siamese embedding performs the worst. With more sentences, the siamese network trained on the general documents is not able to properly recognize the differences between different documents. The scores among different models are more smooth and thus further confirm more data could lead to a more stable result.