Exploiting Global Contextual Information for Document-level Named Entity Recognition

Let $\mathbf{X}_{\mathcal{D}}=\{x_{i}\}_{m}$ be a sequence composed of $m$ tokens that represents a document $\mathcal{D}$ and $\mathbf{Y}_{\mathcal{D}}=\{y_{i}\}_{m}$ be its corresponding sequence of tags over $\mathbf{X}_{\mathcal{D}}$. Specifically, each sentence $s_{i}$ in $\mathcal{D}$ is denoted by $s_{i}=\{x^{i}_{j}\}_{n_{i}}$, where $n_{i}$ is the length of $s_{i}$ and $x^{i}_{j}$ indicates the $j$-th token of $s_{i}$. Formally, given a document $\mathcal{D}$, the problem of document-level name entity recognition (NER) is to learn a parameterized ($\theta$) prediction function, i.e., $f_{\theta}\colon\mathbf{X}_{\mathcal{D}}\mapsto\mathbf{Y}_{\mathcal{D}}$ from input tokens to NER labels over the entire document $\mathcal{D}$.

In Bi-LSTM-CRF Model, an input sequence $\mathbf{X}=[x_{1},x_{2},\ldots,x_{n}]$ is typically encoded into a sequence of low-dimensional distributed dense vectors, in which each element is formulated as a concatenation of pre-trained $\mathbf{w}_{i}$ and character-level $\mathbf{c}_{i}$ word embeddings $\mathbf{x}_{i}=[\mathbf{w}_{i},\mathbf{c}_{i}]$. Then, a context encoder is employed (i.e., Bi-directional LSTM) to encode the embedded $\mathbf{X}$ into a sequence of hidden states $\mathbf{H}=[\mathbf{h}_{1},\mathbf{h}_{2},\ldots,\mathbf{h}_{n}]$ for capturing the local context dependencies within sequence, namely,

where $\stackrel{{\scriptstyle\rightarrow}}{{\mathbf{h}_{i}}}$ and $\stackrel{{\scriptstyle\leftarrow}}{{\mathbf{h}_{i}}}$ are the hidden states incorporating past and future contexts of $\mathbf{x}_{i}$, respectively.

Subsequently, conditional random field (CRF [16]), which is commonly used as a tag decoder, is employed for jointly modeling and predicting the final tags based on the output (i.e., $\mathbf{H}$) of the context encoder, which can be transformed to the probability of label sequence $\mathbf{Y}$ being the correct tags to the given $\mathbf{X}$, namely,

where $\phi(.)$ is the score function; $\mathbf{Y}(\mathbf{X})$ is the set of possible label sequences for $\mathbf{X}$; $\mathbf{W}_{y_{j-1},y_{j}}$ and $b_{y_{j-1},y_{j}}$ indicate the weight matrix and bias corresponding to $(y_{j-1},y_{j})$. Then, a likelihood function $\mathcal{L}_{m}$ is employed to maximize the negative log probability of ground-truth sentences for training,

where $\mathcal{X}$ denotes the set of training instances, and $\mathcal{Y}$ indicates the corresponding tag set.

Epistemic (model) uncertainty explains the uncertainty in the model parameters. In this paper, we adopt the epistemic uncertainty to find out tokens whose labels are likely to be incorrectly predicted by the model, thereby pruning the document graph to effectively prevent the further spread of noise information.

Bayesian probability theory provides mathematical tools to estimate model uncertainty, but these usually lead to high computational cost. Recent advances in variational inference have introduced new techniques into this field, which are used to obtain new approximations for Bayesian neural networks. Among these, Monte Carlo dropout [9] is one of the simplest and most effective methods, which requires little modification to the original model and is applicable to any network architecture that makes use of dropout.

Given the dataset $\mathcal{D}$ with training inputs $\mathbf{X}=\{x_{1},\ldots,x_{n}\}$ and their corresponding labels $\mathbf{Y}=\{y_{1},\ldots,y_{n}\}$, the objective of Bayesian inference is estimating $p(\mathbf{w}|\mathcal{D})$, i.e., the posterior distribution of the model parameters $\mathbf{w}$ given the dataset $\mathcal{D}$. Then the prediction for a new input $x^{*}$ is obtained as follows:

Since the posterior distribution $p(\mathbf{w}|\mathcal{D})$ is intractable, Monte Carlo dropout adopts the variational inference approach to use $q_{\theta}(\mathbf{w})$, a distribution over matrices whose columns are randomly set to zero, to approximate the intractable posterior. Then the minimisation objective is the Kullback-Leibler (KL) divergence between the approximate posterior $q_{\theta}(\mathbf{w})$ and the true posterior $p(\mathbf{w}|\mathcal{D})$.

With Monte Carlo sampling strategy, the integral of Eq. 7 can be approximated as follows:

where $\mathbf{w}_{t}\sim q_{\theta}(\mathbf{w})$ and $T$ is the number of sampling. In fact, this is equivalent to performing $T$ stochastic forward passes through the network and averaging the results. In contrast to standard dropout that only works during training stage, the Monte Carlo dropouts are also activated at test time. And the predictive uncertainty can be estimated by collecting the results of stochastic forward passes through the model and summarizing the predictive variance.

In this section, we focus on fully exploiting the document-level global contexts for enhancing the word-level representation capability of tokens, and thus propose a novel NER model named GCDoc, and then we proceed to structure the following sections with more details about each submodule of GCDoc, which is under the Bi-LSTM-CRF framework (rf. Section III-B) and shown in Figure 2.

Generally, the adjacent sentences may be naturally topically relevant to the current sentence within a same document, and thus we propose a novel cross-sentence contextual embedding module (bottom part in Figure 2, rf. Section IV-B) for encoding the global-level context information at sentence-level to enhance the representations of tokens within the current sentence.

For enriching the contextual representation of each token, we develop a document graph module based on gated graph neural networks to capture the non-local dependencies between tokens across sentences, namely,

where $GGNN(.)$ is the output of graph neural network for learning the contextualized representations of tokens. Then, the output of Eq (10) can be fed into the tag decoder for training via Eq (4), Eq (5) and Eq (6).

Generally, the adjacent sentences may be topically relevant to the current sentence within a same document naturally, and thus we propose a novel cross-sentence contextual embedding module. To this end, in this section we propose a cross-sentence module, which treats the adjacent sentences as auxiliary context at sentence-level to enhance the representation capability of the given sentence. Similar to [26], we only take account of the cross-sentence context information within a certain range (e.g., $k$) before and after the current sentence, namely, we concatenate the $k$ sentences before/after the current sentence for capturing cross-sentence context information.

Without loss of generality, suppose the representations of the current sentence, the past context representation of sentences and the future context representation of sentences are denoted by $\mathbf{s}_{c}$, $\mathbf{s}_{p}$ and $\mathbf{s}_{f}$, respectively. Furthermore, we also introduce an attention mechanism for measuring the relevance of such two auxiliary representations to the current one, and thus a weighted sum ($\tilde{\mathbf{s}}_{c}$) of such two representations is calculated for the representation of the current sentence, namely,

where $a_{p}$ and $a_{f}$ are the learned attentions of the past and the future context information to the current sentence, which are used to control the contributions of such two context information and computed based on softmax,

where $f(\mathbf{s}_{i},\mathbf{s}_{j})$ is a compatibility function to calculate the pair-wise similarity between two sentences, which is computed by

where $\sigma(\cdot)$ is an activation function; $\mathbf{W_{s}^{(i)}}$, $\mathbf{W_{s}^{(j)}}\in\mathbb{R}^{d_{s}\times{d_{s}}}$ are the learnable weight matrices; $\mathbf{w}^{\top}\in\mathbb{R}^{d_{s}}$ is a weight vector, and $\mathbf{b}$ denotes the bias vector.

Finally, a gating mechanism is employed to decide the amount of cross-sentence context information ($\tilde{\mathbf{s}}_{c}$) that should be incorporated and fused with the initial sentence representation $\mathbf{s}_{c}$:

where $\odot$ represents elementwise multiplication, and $\lambda$ is the trade-off parameter, which is calculated by

where $\mathbf{W_{g}^{(1)}},\mathbf{W_{g}^{(2)}},\mathbf{W_{g}^{(3)}}\in\mathbb{R}^{d_{s}\times{d_{s}}}$ are trainable weight matrices.

As mentioned, an intuition is that the entity mentions appearing repeatedly in a document are more likely to be of same entity category. To this end, in this section, we develop a simple but effective graph layer based on a word-level graph, which is built over the re-occurrent tokens to capture the non-local dependencies among them.

Specifically, for each node $v_{i}$, the information propagation process between different nodes can be formalized as:

where $\mathcal{N}(i)$ denotes the set of neighbors of $v_{i}$, which does not include $v_{i}$ itself; $\mathbf{W}_{a}\in\mathbb{R}^{d\times d}$ indict the weight matrices and $\mathbf{b}\in\mathbb{R}^{d}$ denotes the bias vector; $\mathbf{h}_{j}$ is the hidden state of the neighbor node $v_{j}\in\mathcal{N}(i)$.

Through Eq (18), we can easily aggregate the neighbours’ information to enrich the context representation of node $v_{i}$ via the constrained edges ($e_{ij}\in\mathcal{E}_{\mathcal{D}}$). Inspired by the gated graph neural networks, the enhanced representation $\mathbf{a}_{i}$ is combined with its initial embedding ($\mathbf{h}_{i}$) via a gating mechanism, which is employed for deciding the amount of aggregated context information to be incorporated, namely,

where $\mathbf{W}_{z},\mathbf{W}_{r},\mathbf{W}_{o},\mathbf{U}_{z},\mathbf{U}_{r},\mathbf{U}_{o}\in\mathbb{R}^{d\times d}$ are learnable weight matrices, $\sigma(\cdot)$ denotes the logistic sigmoid function and $\odot$ represents elementwise multiplication. $\mathbf{z}_{i}$ and $\mathbf{r}_{i}$ are update gate and reset gate, respectively. The output of Eq (19) $\hat{\mathbf{h}}_{i}$ is the final output of our document graph module (rf. Eq (10)), which will be fed into the tag decoder for generating the tag sequence.

Furthermore, directly aggregating the contextual information may contain noise, since not all re-occurrent tokens can provide reliable context information for disambiguating others. As such, we propose two strategies to avoid introducing noise information in the document graph module. First we apply the epistemic uncertainty to find out tokens whose labels are likely to be incorrectly predicted, thereby pruning the document graph. Then we adopt a selective auxiliary classifier for distinguishing the entity categories of two different nodes, thus guiding the process of calculating edge weights.

As for the model uncertainty prediction, we apply the Monte Carlo dropout and adopt a independent sub-module that predicts labels of each token by a simple architecture composed of a Bi-LSTM and dense layer. The forward pass of the Bi-LSTM is run $T$ times with the same inputs by using the approximated posterior $q_{\theta}(\mathbf{w})$ in Eq 8. Then given the output representation $\mathbf{h}_{i}$ of the Bi-LSTM layer, the probability distribution of the $i$-th word’s label can be obtained by a fully connected layer and a final softmax function,

where $\mathbf{w}_{t}\sim q_{\theta}(\mathbf{w})$, $\mathbf{W}\in\mathbb{R}^{d\times\mathcal{C}}$, and $\mathcal{C}$ is the number of all possible labels. Then, the uncertainty value of the token can be represented by the uncertainty of its corresponding probability vector $\mathbf{p}_{i}$, which can be summarized using the entropy of the probability vector:

In specific, the loss function of the selective auxiliary classifier is defined as:

where $y_{ij}$ is the ground truth label whose value is assigned $1$ when nodes $v_{i}$ and $v_{j}$ belong to the same entity category, and $0$ for otherwise. As a result, Eq (23) can be transformed as follows via the computed relation scores, namely,

After obtaining the enhanced representation $\mathbf{a}_{i}$ that encodes the neighbours’ information, Eq (19)-(22) are applied to generate the final output of our document graph module $\hat{\mathbf{h}}_{i}$.

The employed auxiliary classifier can be regarded as a special regularization term, which explicitly injects the supervision information into the process of calculating edge weights and helps the model select useful information in the process of aggregating neighbors.

The final loss of our model is the sum of loss $\mathcal{L}_{m}$ (Eq 6) and loss $\mathcal{L}_{a}$ (Eq 27),

where $\theta$ is a hyper-parameter.

We use two benchmark NER datasets for evaluation, i.e., CoNLL 2003 dataset (CoNLL03) and OntoNotes 5.0 English datasets (OntoNotes 5.0). The details about corporas are shown in Table I.

• •

  CoNLL03 [33] is a collection of news wire articles from the Reuters corpus, which includes four different types of named entities: persons (PER), locations (LOC), organizations (ORG), and miscellaneous (MISC). We use the standard dataset split [7] and follow BIOES tagging scheme (B, I, E represent the beginning, middle and end of an entity, S means a single word entity and O means non-entity).

• •

  OntoNotes 5.0 is much larger than CoNLL 2003 dataset, and consists of text from a wide variety of sources (broadcast conversation, newswire, magazine and Web text, etc..) It is tagged with eighteen entity types, such as persons (PERSON), organizations (ORG), geopolitical entity (GPE), and law (LAW), etc.. Following previous works [5], we adopt the portion of the dataset with gold-standard named entity annotations, in which the New Testaments portion is excluded.

In this section, we show the details of network training. Related hyper-parameter settings are presented including initialization, optimization, and network structure. We also report the training time of our model on the two datasets.