Modeling Task Interactions in
Document-Level Joint Entity and Relation Extraction

Our first setting is the pipeline approach that trains COREF and RE models separately, and decodes in the naive pipeline manner, where the extracted entities (entity clusters) are first obtained by the COREF model, and then fed to the RE model that produces the final relation triples.

Our second setting features the common joint paradigm adopted in most related work Luan et al. (2019); Zaporojets et al. (2021); Eberts and Ulges (2021) that shares the same encoder and mention representation for all tasks, while keeping independent decoders for COREF and RE that are jointly trained in a multi-task manner (adding two losses). This and later settings employ “shared representation” as the first type of task interactions.

As the COREF model operates on the mention-level but ATLOP scores between entities directly, we propose another joint model that unifies all scoring on the mention-level, allowing more straightforward inter-task interference later.

Same as the baseline, the COREF module in Joint-M still generates a set of mention candidates $(m_{1},..,m_{n})$ and their pairwise coreference scores $s^{c}(m_{x},m_{y})$ indexed by $x,y\in[1,n]$. Different from ATLOP that obtains entity representation first and performs relation scoring among entities, the RE module in Joint-M simply obtains mention-level pairwise relation scores $s^{r}$ through a lightweight biaffine scoring, directly on the same set of mention candidates. More formally:

	$\displaystyle s^{c}(m_{x},m_{y})$	$\displaystyle=g_{x}W^{c}g_{y}^{T}+s^{m}(g_{x})+s^{m}(g_{y})$
	$\displaystyle s^{r_{i}}(m_{h},m_{t})$	$\displaystyle=g_{h}W^{r_{i}}g_{t}^{T}+s^{h_{i}}(g_{h})+s^{t_{i}}(g_{t})$

$g$ denotes the embedding of the corresponding mention; $W^{c}$/$W^{r_{i}}$ are learned parameters for COREF scoring and RE scoring of the $i$th relation type. $s^{m}$/$s^{h_{i}}$/$s^{t_{i}}$ are additional prior scores predicted by separate feed-forward networks on how likely the mention span is a gold mention ($s^{m}$) or a head/tail mention for the $i$th relation type ($s^{h_{i}}$/$s^{t_{i}}$).

Though the original relation labels are on the entity-level, we transfer the labels to the mention-level by letting any mention pair $(m_{h},m_{t})$ express the same relations as their belonging entities $(e_{h},e_{t})$, with $m_{h}\in e_{h}$ and $m_{t}\in e_{t}$. By doing so, the model is forced to learn more inter-sentence reasoning implicitly in the encoding stage to aggregate different local context of mentions belonging to the same entity. Similar mention-level decoding is also adopted in previous work Zaporojets et al. (2021); Eberts and Ulges (2021). In particular, Eberts and Ulges (2021) applies multi-instance learning on mentions; nevertheless, their approach regards mention-level labels as latent variables and still needs to formulate the entity representation, while Joint-M offers a simpler paradigm that discards entities in the model completely, and yields similar performance as multi-instance learning in preliminary experiments.

Joint-M is trained similar to Joint and still employs the same task interaction as “shared representation”. For inference, we obtain the entity-level relation labels by simply averaging the mention-level relation scores from the cartesian product of the predicted entity clusters, denoted as $s^{r_{i}}(e_{h},e_{t})=$ MEAN{$s^{r_{i}}(m_{h},m_{t})$}, $\forall(m_{h},m_{t})\in e_{h}\times e_{t}$.

In this setting, we apply Graph Propagation upon Joint-M, which has the similar formulation as DyGIE++ (Wadden et al., 2019). Distinguished from the original DyGIE++ that only extracts intra-sentence relations, we use our adapted version for the document-level graph propagation as follows.

After the RE scoring in Joint-M, we regard each mention candidate as a graph node and their relation scores as weighted graph edges. Instead of propagating on one graph as DyGIE++, each relation type inherently forms its own directed subgraph that only consists of edges of a specific type. In +GP, we perform subgraph propagation respectively, and then obtain the final node representation by aggregating nodes from each subgraph.

More formally, let $R$ be the set of relation types. $|R|$ heterogeneous relation subgraphs can thus be constructed after the RE scoring. We then apply Graph Attention Network (GAT)-like propagation (Veličković et al., 2018) on each subgraph:

	$\displaystyle\alpha^{r_{i}}_{ht}$	$\displaystyle=\frac{\exp\big{(}\text{ReLU}\big{(}s^{r_{i}}(m_{h},m_{t})\big{)}\big{)}}{\sum_{k\in\mathcal{N}_{h}}\exp\big{(}\text{ReLU}\big{(}s^{r_{i}}(m_{h},m_{k})\big{)}\big{)}}$		(1)
	$\displaystyle g^{r_{i}}_{h}$	$\displaystyle=\tanh(\sum_{t\in\mathcal{N}_{h}}\alpha^{r_{i}}_{ht}\cdot g_{t}W^{r_{i}})$		(2)
	$\displaystyle\hat{g}_{t}$	$\displaystyle=g_{t}+\sum_{r_{i}\in R}g^{r_{i}}_{h}/|R|$		(3)

$\hat{g}_{t}$ is the new tail embedding after the propagation that will replace $g_{t}$; $\mathcal{N}_{h}$ is the set of neighboring nodes of $m_{h}$, which in this case are all the mention candidates. $W^{r_{i}}$ is the learned matrix for type-specific node transformation. The new head embedding $\hat{g}_{h}$ will also be obtained accordingly.

With the new node embedding that fuses the RE decisions, +GP performs the COREF scoring as in Joint-M but using the updated mention representation, accomplishing implicit task interactions. We do not perform further propagation on COREF graphs as it is shown little effects by previous work (Wadden et al., 2019; Xu and Choi, 2020).

As above interactions are all implicit, we propose to leverage task characteristics between COREF and RE to design explicit task interactions, dubbed Graph Compatibility as a new setting upon Joint-M. Specifically, each node after RE scoring can be regarded as a local graph that connects to all other nodes with weighted edges (relation scores). If two mention nodes are from the same entity cluster, their local graphs should be similar, since they are forced by Joint-M to have the exact same relations to other nodes; vice versa, if two nodes do not refer to the same entity, their relations (weighted edges) to other mentions are likely to be distant from each other. Therefore, our +GC model learns a distance metric to check the “compatibility” of local relation graphs, as an additional clue of how likely two mentions are coreferent.

More formally, this second source of coreference scores $\hat{s}^{c}$ can be denoted as:

	$\displaystyle d^{r_{i}}_{x,y}=\sum_{k\in\mathcal{N}_{x,y}}$	$\displaystyle|s^{r_{i}}(m_{x},m_{k})-s^{r_{i}}(m_{y},m_{k})|$		(4)
	$\displaystyle\hat{s}^{c}(m_{x},m_{y})$	$\displaystyle=\sum_{r_{i}\in R}\beta^{r_{i}}\cdot d^{r_{i}}_{x,y}$		(5)
	$\displaystyle\widetilde{s}^{c}(m_{x},m_{y})$	$\displaystyle=s^{c}(m_{x},m_{y})-\lambda\hat{s}^{c}(m_{x},m_{y})$

$d^{r_{i}}_{x,y}$ is the raw L1 distance between the two local graphs by all neighboring edges of the $r_{i}$ relation type. $\hat{s}^{c}$ is the final distance/compatibility of two local graphs, weighted by the learned parameter $\beta^{r_{i}}$ that determines the importance of each $r_{i}$; higher $\hat{s}^{c}$ indicates more diverging graphs. The final coreference score $\widetilde{s}^{c}$ interpolates the original $s^{c}$ and the new distance $\hat{s}^{c}$, with $\lambda$ being a hyperparameter.

Overall, +GC enables explicit interactions that bridge COREF and RE together: RE can affect COREF directly, while COREF also pushes similar RE scores for coreferent pairs during back-propagation. The final distance $\hat{s}^{c}$ is optimized by a contrastive loss as in Eq (6) that is commonly used in Siamese Network (Koch et al., 2015). For simplicity, denote $D=\hat{s}^{c}(m_{x},m_{y})$, $Y=1$ when $(m_{x},m_{y})$ is from the same entity, and $Y=0$ elsewise. $m$ is the margin as a hyperparameter. $\hat{\mathcal{L}}$ is added as the third loss in Joint-M’s training.

$\displaystyle\hat{\mathcal{L}}=Y\cdot D^{2}+(1-Y)\cdot\max(0,m-D)^{2}$

(6)

As the relation graphs are inevitably sparse because only a small fraction of mention pairs express relations, we reduce the overhead introduced by $k$ in Eq (4) by pruning the local graphs based on heuristics described in Appendix A.2.