Entity Structure Within and Throughout: Modeling Mention Dependencies
for Document-Level Relation Extraction

Entity structure describes the distribution of entity instances over texts and the dependencies among them. In the specific scenario of document-level texts, we consider the following two structures.

• •

  Co-occurrence structure: Whether or not two mentions reside in the same sentence.
  

• •

  Coreference structure: Whether or not two mentions refer to the same entity.
  

Both structures can be described as True or False. For co-occurrence structure, we segment documents into sentences, and take them as minimum units that exhibit mention interactions. So True or False distinguishes intra-sentential interactions which depend on local context from inter-sentential ones that require cross sentence reasoning. We denote them as intra and inter respectively. For coreference structure, True indicates that two mentions refer to the same entity and thus should be investigated and reasoned with together, while False implies a pair of distinctive entities that are possibly related under certain predicates. We denote them as coref and relate respectively. In summary, these two structures are mutually orthogonal, resulting in four distinctive and undirected dependencies, as shown in table 1.

Besides the dependencies between entity mentions, we further consider another type of dependency between entity mentions and its intra-sentential non-entity (NE) words. We denote it as intraNE. For other inter-sentential non-entity words, we assume there is no crucial dependency, and categorize it as NA. The overall structure is thus formulated into an entity-centric adjacency matrix with all its elements from a finite dependency set: {intra+coref, inter+coref, intra+relate, inter+relate, intraNE, NA} (see figure 2).

SSAN inherits the architecture of Transformer (Vaswani et al. 2017) encoder, which is a stack of identical building blocks, wrapped up with feedforward network, residual connection, and layer normalization. As its core component, we propose structured self-attention mechanism with two alternative transformation modules.

Given an input token sequence $x=(x_{1},x_{2},...,x_{n})$, following the above formulation, we introduce $S=\{s_{ij}\}$ to represent its structure, where $i,j\in\{1,2,...,n\}$ and $s_{ij}\in${intra+coref, inter+coref, intra+relate, inter+relate, intraNE, NA} is a discrete variable denotes the dependency from $x_{i}$ to $x_{j}$. Note that here we extend dependency from mention-level to token-level for practical implementation. If mention instance consists of multiple subwords ($E3$ in figure 2, $S2$), we assign dependencies for each token accordingly. Within each mention, subword pairs should conform with intra+coref and thus are assigned as such.

In each layer $l$, the input representation $\boldsymbol{x}_{i}^{l}\in\mathbb{R}^{d_{in}}$ is first projected into query / key / value vector respectively:

$$\boldsymbol{q}_{i}^{l}=\boldsymbol{x}_{i}^{l}\boldsymbol{W}_{l}^{Q},\boldsymbol{k}_{i}^{l}=\boldsymbol{x}_{i}^{l}\boldsymbol{W}_{l}^{K},\boldsymbol{v}_{i}^{l}=\boldsymbol{x}_{i}^{l}\boldsymbol{W}_{l}^{V}$$

(1)

where $\boldsymbol{W}_{l}^{Q}$,$\boldsymbol{W}_{l}^{K}$,$\boldsymbol{W}_{l}^{V}\in\mathbb{R}^{d_{in}\times d_{out}}$. Based on these inputs and entity structure $S$, we compute unstructured attention score and structured attentive bias, and then aggregate them together to guide the final self-attention flow.

The unstructured attention score is produced by query-key product as in standard self-attention:

$$e_{ij}^{l}=\frac{\boldsymbol{q}_{i}^{l}{\boldsymbol{k}_{j}^{l}}^{T}}{\sqrt{d}}$$

(2)

Parallel to it, we employ an additional module to model the structural dependency conditioned on their contextualized query / key representations. We parameterize it as transformations which project $s_{ij}$ along with query vector $\boldsymbol{q}_{i}^{l}$ and key vector $\boldsymbol{k}_{j}^{l}$ into attentive bias, then impose it upon $e_{ij}^{l}$:

$$\tilde{e}_{ij}^{l}=e_{ij}^{l}+\frac{transformation(\boldsymbol{q}_{i}^{l},\boldsymbol{k}_{j}^{l},s_{ij})}{\sqrt{d}}$$

(3)

The proposed transformation module regulates the attention flow from $x_{i}$ to $x_{j}$. As a consequence, the model benefits from the guidance of structural dependencies.

After we obtain the regulated attention scores $\tilde{e}_{ij}^{l}$, a softmax operation is applied, and the value vectors are aggregated accordingly:

$$\boldsymbol{z}_{i}^{l+1}=\sum_{j=1}^{n}\frac{exp~{}\tilde{e}_{ij}^{l}}{\sum_{k=1}^{n}exp~{}\tilde{e}_{ik}^{l}}\boldsymbol{v}_{j}^{l}$$

(4)

here $\boldsymbol{z}_{i}^{l+1}\in\mathbb{R}^{d_{out}}$ is the updated contextual representation of $\boldsymbol{x}_{i}^{l}$. Figure 2 gives the overview of SSAN. In the next section, we describe the transformation module.

To incorporate the discrete structure $s_{ij}$ into an end-to-end trainable deep model, we instantiate each $s_{ij}$ as neural layers with specific parameters, train and apply them in a compositional fashion. As a result, for each input structure $S$ composed of $s_{ij}$, we have a structured model composed of corresponding layer parameters. As for the specific design of these neural layers, we propose two alternatives: Biaffine Transformation and Decomposed Linear Transformation:

$$\begin{split}bias_{ij}^{l}&=Biaffine(s_{ij},\boldsymbol{q}_{i}^{l},\boldsymbol{k}_{j}^{l})\\ &or\\ &=Decomp(s_{ij},\boldsymbol{q}_{i}^{l},\boldsymbol{k}_{j}^{l})\end{split}$$

(5)

The proposed SSAN model takes document text as input, and builds its contextual representations under the guidance of entity structure within and throughout the overall encoding stage. In this work, we simply use it for relation extraction with minimum design. After the encoding stage, we construct a fixed dimensional representation for each target entity via average pooling, which we denote as $\boldsymbol{e}_{i}\in\mathbb{R}^{d_{e}}$. Then, for each entity pair, we compute the probability of relation $r$ from the pre-specified relation schema as:

$$P_{r}(\boldsymbol{e}_{s},\boldsymbol{e}_{o})=sigmoid(\boldsymbol{e}_{s}\boldsymbol{W}_{r}\boldsymbol{e}_{o})$$

(9)

where $\boldsymbol{W}_{r}\in\mathbb{R}^{d_{e}\times d_{e}}$. The model is trained using cross entropy loss:

$$L=\sum_{<s,o>}\sum_{r}CrossEntropy(P_{r}(\boldsymbol{e}_{s},\boldsymbol{e}_{o}),\overline{y}_{r}(\boldsymbol{e}_{s},\boldsymbol{e}_{o}))$$

(10)

and $\overline{y}$ is the target label. Given $N$ entities and a relation schema of size $M$, equation 9 should be computed $N\times N\times M$ times to give all predictions.

We evaluate the proposed approach on three popular document-level relation extraction datasets, namely DocRED (Yao et al. 2019), CDR  (Li et al. 2016a) and GDA  (Wu et al. 2019), all involving challenging relational reasoning over multiple entities across multiple sentences. We summarize their information in Appendix A.

We initialize SSAN with different pretrained language models including BERT (Devlin et al. 2019), RoBERTa (Liu et al. 2019) and SciBERT (Beltagy, Lo, and Cohan 2019).

On each dataset, we give comprehensive results of SSAN initialized with different pretrained language models along with their corresponding baselines for fair comparisons. The parameters in newly introduced transformation modules are learned from scratch. All results are obtained using grid search for hyper-parameters (see appendix B for detail) on the development set, then the best model is selected to produce results on the test set. On DocRED, following the official baseline implementation (Yao et al. 2019), we utilize naive features including entity type and entity coreference, which is added to the input word embedding. We also concatenate entity relative distance embedding of each entity pair before the final classification. We preprocess CDR and GDA dataset following Christopoulou, Miwa, and Ananiadou (2019). On CDR, after the best hyper-parameter is set, we merge the training set and dev set to train the final model, on GDA, we split 20% of the training set for development.

We conduct comprehensive and comparable experiments on DocRED dataset. We report both F1 and Ign F1 according to Yao et al. (2019). Ign F1 is computed by excluding relational facts that already appeared in the training set.

As shown in table 2, SSAN with both Biaffine and Decomp transformation can consistently outperform their baselines with considerable margin. In most of the results, Biaffine brings more considerable performance gain compared to Decomp, which demonstrates that the former is of greater ability to model structural dependencies.

We compare our model with previous works that either do not consider entity structure or do not explicitly model them within and throughout encoders. Specifically, ContexAware (Yao et al. 2019), BERT Two-Phase (Wang et al. 2019a) and HINBERT (Tang et al. 2020) do not consider the structural dependencies among entities. EOG (Christopoulou, Miwa, and Ananiadou 2019) and LSR (Nan et al. 2020) utilize graph methods to perform structure reasoning, but only after the BiLSTM or BERT encoder. CorefBERT and CorefRoBERTa (Ye et al. 2020) further pretrain BERT and RoBERTa with a coreference prediction task to enable implicit reasoning of coreference structure. Results in table 2 shows that SSAN performs better than these methods. Our best model, SSAN_Biaffine built upon RoBERTa Large, is +2.41 / +1.79 Ign F1 better on dev / test set than CorefRoBERTa Large (Ye et al. 2020), and +1.80 / +1.04 Ign F1 better than our baseline. In general, these results demonstrate both the usefulness of entity structure and the effectiveness of SSAN.

Although SSAN is well compatible with pretrained Transformer models, there still exists a distribution gap between parameters in newly introduced transformation layers and those already pretrained ones, thus impedes the improvements of SSAN to a certain extent. In order to alleviate such distribution deviation, we also utilize the distantly supervised data from DocRED, which shares identical format with the trainset, to first pretrain SSAN before finetuning on the annotated training set for better adaptation. Here we choose our best model, SSAN_Biaffine built upon RoBERTa Large, and denote it as +Adaptation in table 2 (see appendix B for hyperparameters setting). The resulting performance are greatly improved, achieving 63.78 Ign F1 and 65.92 F1 on test set as well as the 1st position on the leaderboard³³3https://competitions.codalab.org/competitions/20717#results at the time of submission.

On CDR and GDA datasets, besides BERT, we also adopts SciBERT for its superiority when dealing with biomedical domain texts. On CDR test set (see Table 3), SSAN obtains +1.3 F1/+1.7 F1 gain based on BERT Base/Large and +2.9 F1 gain based on SciBERT, which significantly outperform the baselines and all existing works. On GDA (see Table 4), similar improvements can also be observed. These results demonstrate the strong applicability and generality of our approach.

We perform ablation studies of the proposed approach on DocRED. Again, we consider SSAN_Biaffine built upon RoBERTa Large. Table 5 gives the results of SSAN when each structural dependency is excluded. It is clear that all five dependencies contribute to the final improvements. We can arrive at the conclusion that the proposed entity structure formulation is indeed helpful priors for document-level relation extraction. We can also see that intra+coref effects the most among all dependencies.

We also look into the design of two transformation modules by testing each bias term respectively. As shown in table 6, all bias terms can improve the result over baseline, including the prior bias $+b_{s_{ij}}$ that is only individual values. Among all bias terms, biaffine bias $+\boldsymbol{q}_{i}\boldsymbol{A}_{s_{ij}}\boldsymbol{k}_{j}^{T}$ is the most effective, brings +1.38 Ign F1 improvements solely. For Decomposed Linear Transformation, key conditioned bias $+\boldsymbol{Q}_{s_{ij}}\boldsymbol{k}_{j}^{T}$ produces better results than query conditioned bias $+\boldsymbol{q}_{i}\boldsymbol{K}_{s_{ij}}^{T}$, which implies that the key vectors might be associated with more entity structure information.

As a key feature of SSAN is to formulate entity structure priors into attentive biases, it would be instructive to explore how such attentive biases regulate the propagation of self-attention bottom-to-up. To this purpose, we collect all attentive biases produced by SSAN_Biaffine (built upon RoBERTa Large) for DocRED dev instances, categorized according to dependency types, and averaged across all attention heads and all instances. Figure 3 (a) is the resultant heatmap, where each cell indicates the value of averaged bias at each layer (horizontal axis) for each entity dependency type (vertical axis). We can observe meaningful patterns: 1) Along the horizontal axis, the bias is relatively small at bottom layers, where the self-attention score will be mainly decided by unstructured semantic contexts. It then grows gradually and reaches the maximum at the top-most layers, where the self-attention score will be greatly regulated by the structural priors. 2) Along the vertical axis, at the top-most layers (inside the dotted bounding box), bias from inter+coref is significantly positive. This conforms with human intuition that coreferential mention pairs might act as a bridge for cross-sentence reasoning, thus should enable more information passing. While biases from intra+relate and inter+relate appear in contrast.

Based on the discussion, we further investigate the effect of different layers to impose attentive biases. As shown in Figure 3 (b), with only the top 4 layers (1/6 of the total layers) integrated with entity structure, SSAN can keep +0.89 F1 gain, which confirms that these top-most layers with larger biases indeed impact more significantly. In the meantime, with more layers included, the performance still improves, and reaches the best of +1.50 F1 with all 24 layers equipped with structured self-attention.

We thank all anonymous reviewers for their valuable comments. This work is supported by the National Key Research and Development Project of China (No.2018YFB1004300, No.2018AAA0101900), and the National Natural Science Foundation of China (No.61876223, No.U19A2057).

Table 7 details statistics of entities along with other related information of three selected datasets. We can see that all three datasets entail more than two dozen mentions per document on average, with each sentence contains approximately three mentions on average. These statistics further demonstrate the complexity of entity structure in document-level relation extraction tasks.

Table 8 details our hyper-parameters setting. All experiment results are obtained using grid search on the development set. All comparable results share the same search scope.