Text-guided Legal Knowledge Graph Reasoning

We collect all the data from the Guangdong government service website. We obtain about 140,482 raw affairs (including 1,552 unique affairs), and 4,042 laws with 269,053 legal provisions. We perform detailed analysis and conduct data preprocessing procedures to address those issues below:

Non-standard text. There exist a huge discrepancy for the legal provisions and affairs, including: Abbreviation, such as “劳动法” (Labor Law) is the abbreviation of “中华人民共和国劳动法” (People’s Republic of China Labor Law); Missing, such as missing of angle quotation mark (“《》” ) or the format of the version number (for example, the suffix of the “《广东省民用建筑节能条例》（2014年修正本）” (”Regulations on Energy Conservation of Civil Buildings in Guangdong Province” (Amended in 2014)). Those challenges make it difficult to establish the association between affairs and legal provisions. To handle those non-standard texts, we manually build a legal provision dictionary to normalize those non-standard texts.

Similar affairs. From the raw data, we observe that a huge portion of affairs is very similar (with the same affair vertexes). Statistically, we find that the ratio of unique items to the total number of items is roughly 1:100. We analyze those similar affairs and find that most parts of them have the same content, while only the time in the affair is different. We merge those similar affairs in the prepossessing procedure.

No legal provisions. We observe that several affairs do not have any linking legal provisions, which implies no legal provisions. This problem is mainly due to outdated laws. As the legal provision will change over time, several old provisions may be deleted, making several affairs impossible to link. Also, there exists a little legal provision that does not have standard formats (in general, the standard format of the legal provision is XX law, chapter X, article X, paragraph X); thus, affairs cannot be linked to those legal provisions either. We filter out those no legal provision affairs in the prepossessing procedure.

Our proposed legal task is a real-world application. As the fast updating of affairs and legal provisions, newly added affairs cannot be linked with existing legal provisions. We notice that there exist relations between affairs and laws following a well-defined schema. From two-hop reasoning on the legal knowledge graph, it is possible to judge whether there exists a relation between an affair and legal provisions. In this paper, we formulate the legal provision prediction task as the link prediction problem in the legal knowledge graph. We model the legal provision, affair, law, right as entities, as shown in Figure 1. We detail the statistic of the legal knowledge graph in Table 2.

We randomly divide the triples with four type relations into the train, valid and test set with a ratio of 8:1:1, and we filter the triples with base_entry_is relation from the test set as the target test set. The detailed number of entities, relations, and triples of the LegalLPP dataset are shown in Table 3.

A knowledge graph $G$ is a set of triplets in the form $(h,r,t)$, $h,t\in\mathcal{E}$ and $r\in\mathcal{R}$ where $\mathcal{E}$ is the entity vocabulary and $\mathcal{R}$ is a collection of pre-defined relations as shown in Table 2. We are aimed at predicting whether there exists the relation base_entry_is between affair entities and legal provision entities. We construct positive triples with ground truth instances and negative triples with corrupted instances following [1].

Our text-guided graph reasoning approach consists of two main components, as shown in Figure 2. Our approach is not end-to-end as we firstly fine-tune the text representation and then leverage this feature to perform legal graph reasoning.

Text Representation Learning (§3.3). Given an affair and legal provision, we employ neural networks to encode the instance semantics into a vector. In this study, we implement the instance encoder with BERT [3]. We then apply an MLP layer to reduce the dimension of features (the dimension of BERT-base is 768, which is not convenient for training GNN) to obtain the text representations, which is more efficient for training and inference. We learn the text representation via fine-tuning with triple scores following TransE [1].

Legal Graph Reasoning (§3.4). After obtaining the learned text representations, we employ GNN to learn explicit relational knowledge. By assimilating generic message-passing inference algorithms with the neural-network counterpart, we can learn vertex embeddings with legal reasoning. Then we utilize a residual connection from the text representation to obtain the final representation. Finally, we utilize TransE [1], DistMult [14] and SimplE [5] as triple score functions.

Given an input affair text $h$ and legal provision text $t$, we utilize BERT [3] to obtain the text representations as follows:

where $h$,$t$ are raw input text and $m_{h}$,$m_{t}$ are the output [CLS] embeddings of BERT. We then leverage an MLP layer to reduce dimension as follows:

where $v_{h}$, $w_{t}$ are the final text representations which will then be fed into the GNN. To obtain more representative features, we finetune the text representation with TransE triple score function as:

where $\operatorname{score}(h,r,t)$ is the score of triple $<h,r,t>$, $v_{h}$, $v_{t}$, are entity representations from Eq. 2, $v_{r}$ is random initialized vectors. We further analyze the empirical performance of other triple score function of DistMult and SimplE. The DistMult and SimplE score function are calculated as:

DistMult is a simplified version of RESCAL [9] by using a diagonal matrix to encode relation. Different from DistMult, SimplE can handle asymmetric relations.

We feed the vertex representation $v_{i}$ into a graph encoder to obtain the hidden vectors, which explicitly models the graph structure of the legal knowledge graph. We use an implementation of the GNN model following GAT [13] and R-GCN [11].

GAT. The GAT model uses multiple graph attention layer connections for encoding vertexs. Specifically, GAT calculates the attention weights of neighboring nodes for aggregation. The attention weights of node $i$ and node $j$ are calculated as follows:

where $.T$ represents transposition and $\|$ is the concatenation operation. Once obtained, the normalized attention coefficients are used to compute a linear combination of the features corresponding to them, to serve as the final output features for every node (after potentially applying a nonlinearity, $\sigma$):

where $\vec{v}_{i}^{\prime}$ is the final graph node representation.

R-GCN. R-GCN utilizes multi-layer relational graph convolutional layer to represent node. The forward-pass update of an node denoted by $v_{i}$ in a relational multi-graph is shown as follows:

where $\vec{v}_{i}^{\prime}$ is the final graph node representation, $\mathcal{N}_{i}^{r}$ denotes the set of neighbor indices of node $i$ under relation $r\in\mathcal{R}$. $c_{i,r}$ is a problem-specific normalization constant that can either be learned or chosen in advance (such as $c_{i,r}=\mid\mathcal{N}_{i}^{r}$). Note that, R-GCN considers the relation of triples in the convolution process and is able to learn different aggregation weights according to different relations.

Afterwards, we add a residual connection from the output of MLP layer to the graph node representation, denoted by:

where $v_{i}$ is the final entity representation of entities leveraging both text and graph reasoning.

Finally, we utilize the same score function of TransE, DistMult and SimpLE in the §3.3 to calculate triple scores. Note that, in the graph reasoning stage, $v_{h}$ and $v_{t}$ are combined with both text and graph features while $v_{r}$ is initialized from the tuned embedding in text representation learning (Eq. 3 and Eq. 5). Though our approach is not end-to-end, the entity embeddings (e.g., legal provisions, affairs) can be pre-computed, which is quite efficient in for inference.

We conduct experiments on the LegalLPP dataset. We use Pytorch [10] to implement baselines and our approach on single Nvidia 1080Ti GPU. We leverage Graph Deep Library²²2https://www.dgl.ai/ to implement all the GNN components. We utilize bert-base-Chinese³³3https://github.com/google-research/bert to represent text. We employ Adam [6] as the optimizer. In the text representation learning stage, the learning rate is 5e-5 with the warm-up proportion being 0.1; the batch size is 64, the maximum sequence length of each entity’s text is 128. After 6 epochs of training, we generate 400-dimension text representations. In the legal graph reasoning stage, we set the learning rate of GNN to be 0.01. We train 4,000 epochs for GAT and R-GCN. We use TransE as the default triple score function. We evaluate the performance with Mean Rank (MR), Mean Reciprocal Rank (MRR), and HIT@N (N=1,3,10).

We compare our approach with different kinds of baselines, as shown below:

No reasoning. We conduct TransE [1] as an baseline. We also utilize two separate BERT encoders to represent the text with the TransE triple score function as a baseline.

Graph only. We build the legal knowledge graph and leverage GNN approaches R-GCN and GAT without text features. The graph node representation is initialized randomly.

From Table 4, we observe:

1) Our approach T-GraphR with GAT achieves the best performance. We argue that our target task is to predict the base_entry_is relation between affairs and legal provisions, and there are only four relations in the graph; thus, GAT, which implicitly specifying different weights to different nodes in a neighborhood can obtain better performance.

2) Graph only approach achieves better performance than no reasoning methods BERT and TransE, which indicates that graph reasoning plays a vital role in legal provision prediction.

3) Our T-GraphR approach achieves the best performance and even obtain 12.8% hit@10 improvements compared with the text-only no reasoning model TransE.

4) The overall performance is still far from satisfactory (less than 0.3 with hit@10), and there is more room for future works.

We conduct experiments with different triple score function and report results in Table 4. We observe that TransE obtains better performance than DistMult and SimplE. We argue that the TransE model represents relations as translations, which aims to model the inversion and composition patterns; the DistMult utilizes the three-way interactions between head entities, relations, and tail entities which aims to model the symmetry pattern, the SimplE model the asymmetric relations by considering two (head and tail) vectors only. In our LPP task, those inversion and composition patterns are common in the legal knowledge graph; thus, such translation assumption is advantageous.

We present some predicted instances obtained by our model to demonstrate the generalization ability in Table 5. Our method can predict correct legal provisions with complex surface contexts. Moreover, by reasoning on the legal knowledge graph, we can leverage the well-defined structure, which boosts performance. However, vanilla BERT only considers text, neglecting the structured knowledge in the legal knowledge graph, which results in unsatisfactory performance.

To further analyze the behavior of entity representations, we utilize T-SNE [8] to visualize five randomly selected entity embeddings. From Figure 3, we find that entity embeddings of the graph only approaches have a compact data distribution, while with pre-trained LMs, entities of different types are scattered. To conclude, text features can enhance the vertex’s discriminative ability to enhance node representations.