Trigger-GNN: A Trigger-Based Graph Neural Network for Nested Named Entity Recognition

Graph neural networks have been successfully applied to several text classification tasks [13, 14, 15, 16, 17]. Liu et al. [16] proposed a tensor graph convolutional networks called Tensor-GCN. The Tensor-GCN uses the text graph tensor to describe semantic, syntactic, and sequential contextual information. The model uses the combination of the intra-graph propagation and inter-graph propagation to aggregate the information from the neighborhood nodes in a single graph, and to harmonize heterogeneous information between graphs. Huang et al. [17] proposed to build graph for each input text using sharing parameters, instead of training on a single graph for the whole corpus. It indeed removes the burden of dependence from the individual text to the entire corpus. However, it still preserves the global information. Yao et al. [14] developed a single text graph convolutional network (Text-GCN) based on the word co-occurrence and document-word relations. It jointly learns the embeddings of both words and documents, and it is supervised by the documents’ class labels.

There has been a growing amount of efforts towards NER, and have been explored in several directions, including rule-based, statistics-based and deep neural network-based [1, 9, 18, 6, 19, 20]. However, nested named entity recognition is always a thorny issue because of its complex hierarchical structure.

Towards alleviating the nested NER, recent works have mainly focus on stacking flat NER layers [4, 5, 6, 7]. Ju et al. [5] proposed to recognize the nested entities by stacking flat NER layers dynamically. The model concatenates the output of each LSTM structure in the current flat NER layer to build a revolutionary representation for detected entities, and subsequently feeds them into the next flat NER layer. It makes full use of the encoded information in the corresponding internal entities in an inside-out manner. However, the layered model cannot reuse the outside information, which means it has a single direction for information transfer from the inner layer to the outside way.

Another line of research aims to combine the bipartite graph with the flat NER layers. Luo et al. [8] proposed a bipartite flat GNN to learn the flat entities and their inner dependencies jointly. It constructs a graph module to deliver the bidirectional information from innermost graph layer to the outermost one. The leaned information carries the dependencies of inner entities and can be exploited to improve outermost entity predictions. Due to the graph module, the transfer of the information is bidirectional. The layered model can reuse the outside information as well. However, this method lacks clear and confident rationales to conduct the process of notation. While the entity triggers can bring the complementary annotation to help promote the labelling process.

Inspired by the achievements aforementioned (e.g., graph neural networks on texts [10] and learning with entity triggers [12]), we propose to use a graph neural network integrating the complementary annotation. The Trigger-GNN enables two significant issues: (1) multiple graph-based interactions among the words, entity triggers, and the whole sentence semantics; and (2) using trigger-based instead of lexicon-based [10] can bring some complementary information as the supervision signals, and thus help the model to learn and generalize more efficiently.

In this section, we propose to train an encoder architecture for entity trigger learning, and to match the entity trigger with its corresponding sentence using an attention-based representations. Our intuition is that the desired representation of the entity triggers should involve the semantics with the hidden states of the tokens from the sentence in a shared embedding space. Specifically, for a sentence $S=w_{1},w_{2},\dots,w_{n}$ with multiple entities $e_{1},e_{2},\dots,e_{k}$, we assume that there is a set of triggers $T_{i}=t_{1},t_{2},\ldots,t_{n}$. We reformat the input of our model to enable efficient batch training. Each entity is linked to one of its corresponding trigger denoted as $(x,e_{i},t_{j})$, where $x$ is the tokenization list of the sentence $S$. For each reformed training batch, we first apply a bidirectional LSTM on the sequence of word vectors of $x$ using Glove word embedding [22]. It returns the hidden states $h_{i}$ of each token $x_{i}$, and $\mathbf{H}$ is denoted as the matrix containing the embedding representation for all the tokens; $\mathbf{Z}$ is denoted as the matrix containing the embedding representation for all the trigger $T$. We utilize the self-attention method introduced by [12] to obtain the representation of both triggers and sentences:

where $W_{1}$ and $W_{2}$ are two trainable parameters in the model. $\mathbf{g}_{\mathbf{s}}$ is denoted as the final sentence’s representation. It reveals the weighted sum of all the token representations in the whole sentence. Similarly, $\mathbf{g}_{\mathbf{t}}$ is denoted as the final trigger’s representation.

We train the trigger encoder with the help of the type of the trigger’s associated entity. We use a simple multi-class classifier to predicate the type of such entity, and the corresponding type of each entity returned is denoted as $\Omega$. The loss function of the classifier is as follows:

where $\theta_{1}$ is a trainable parameter in the model.

We match the triggers and sentences by $\mathbf{g}_{\mathbf{t}}$ and $\mathbf{g}_{\mathbf{s}}$ according to Eq.4. During the training process, we mix the triggers and the sentences randomly to sample some negative samples to tackle the imbalance of the positive and negative samples. We expect a margin $m$ between the sample embeddings for the negative situation. The loss function of the semantic matching is as follows:

where $\alpha$ is defined to confirm whether the trigger is originally in the sentence of not. When $\alpha$ is set to $1$, the trigger is originally in the sentence; and when it is set to $0$, the trigger is not originally in the sentence. The joint loss function of the trigger encoding and semantic matching is $L=L1+\lambda L2$, where $\lambda$ is a hyper-parameter for fine-tuning. In our experiment, we define $\lambda=1.3$.

In this section, we detail how to convert the whole sentences into a directed graph. We define each word in the sentences as a node, and add the edges between each node according to the corresponding lexicon³³3To this end, we have to maintain a lexicon list during our experiment, find more details in the Experiment Analysis Section. Also, we design a graph-level node to gather all the information from the nodes and edges in the text graph. The graph-level node can help the node representation by removing the ambiguity.

Formally, let $S=w_{1},w_{2},\dots,w_{n}$ denote a sentence, where $w_{i}$ denotes the $i$-th word. The potential lexicon $T=t_{1},t_{2},\dots,t_{n}$ matching a word sub-sequence can be formulated as $t_{b,e}=w_{b},w_{b+1},\dots,w_{e-1},w_{e}$, where the index of the first word and the last word are $b$ and $e$. In this work, we propose to denote directed and labeled multi-graphs as $G=(V,E,U)$ with nodes (words: $v_{i}\in V$), labeled edges (relations: ($t_{b,e},r)\in E$, where $r\in R$ is a relation type according to the entity trigger), and graph-level node (global attributes: $u_{i}\in U$). Once a word sub-sequence matches a candidate lexicon such as $t_{b,e}$, we add one edge $e_{b,e}\in E$, indicating that it starts from the beginning word $w_{b}$ to the ending word $w_{e}$. The graph-level node is to capture the global information of the entire text graph. Formally, it is represented as the sum of the representation of all the nodes and edges of the graph. For a graph with $n$ word nodes and $m$ edges, there are $m+n$ relations linking each node and edge to the shared graph-level node representation. In addition, we construct the transpose of the text graph $G$ according to [10]. It is another directed graph denoted as $G^{\mathrm{T}}$ with the same set of nodes but all edges reversed compared to $G$. Similar to the bi-directional LSTM, we compose $G$ and $G^{\mathrm{T}}$ as a bi-directional text graph as the input and concatenate the hidden states from $G$ and $G^{\mathrm{T}}$ as the final outputs.

In this part, we detail the structure of the recursive-based graph neural network from three subsections: updation module, aggregation module, and trigger-enhanced decoding and tagging.

We first introduce the NER datasets used in the experiments. All the datasets are public and popular in NER task.

• •

  CoNLL-2002 & CoNLL-2003: We use the English corpora from the CoNLL 2002 and 2003 shared task: language-independent named entity recognition [23, 24] with standard split.

• •

  JNLPBA: We use a molecular biology dataset for identifying the technical terms [25]. It contains various type of nested entities. Examples of such entities include the names of proteins, genes and their location of activity such as cells or organism. We use the same standard split according to [26].

• •

  BC5CDR: Another bio-medical domain dataset which is well-studied and popular in evaluating the performance of nested-NER [27]. We use the standard split.

Both JNLPBA and BC5CDR datasets involves a large number of nested entities in the bio-medical domain.

We use the lexicon generated on a corpus of article pairs from Gigaword ⁴⁴4https://catalog.ldc.upenn.edu/LDC2003T05, consisting around 4 millon articles. Use tensorflow as implementation ⁵⁵5https://www.tensorflow.org/datasets/catalog/gigaword. The embeddings of the lexicon words were pre-trained by Glove word embedding [22] and fine-tuned during training.

Several state-of-the-art NER algorithms (sorted by the timeline) for effectiveness evaluation are listed below:

• •

  Neural layered model: It merged the output of the stacked LSTM layers to build new representation for detected entities. [5].

• •

  Lattice LSTM: It raised a lattice-structure based LSTM model, which encodes a sequence of input characters as well as all the potential words matching a lexicon [11].

• •

  Boundary-aware neural model: It proposed a boundary-aware neural model for nested NER using sequence labelling [28].

• •

  Biaffine-NER: It proposed a graph-based dependency parsing to provide a global view on the sentence [29].

• •

  BiFlaG: It firstly used the entities recognized by the flat NER module to construct an entity graph [8].

• •

  LGN: It developed a lexicon-based graph network with global semantics and proposed a schema to connect the character using external lexicon words [10].

• •

  Trigger-NER: It first introduced the concept of “entity trigger” and proposed a RNN-based trigger matching network for NER [12].

• •

  ACE $+$ document-context: It proposed an automated concatenation method of embeddings (ACE) to automate the process of finding better concatenations of embeddings for structured prediction tasks, NER is one of them [30].

We demonstrate the performance of our Trigger-GNN on both general and bio domains, especially focus on the nested-NER. Results on the CoNLL-2002 & 2003, JNLPBA and BC5CDR datasets are shown in Table I, II, and III, respectively. Compared with the recent methods, our Trigger-GNN obtains the best results by a large margin. In particular, Trigger-GNN obtains 1.93% and 3.52% boosting than its baseline model LGN on the general domain dataset CoNLL-2002 & 2003. And as shown in Table II, and III, Trigger-GNN obtains 3.41% improvement on JNLPBA than LGN, and 5.32% on BC5CDR. The observation from the main results on these four dataset demonstrates that our proposed model can adapt to the both general domain and bio domain, and perform better than the recent methods.