MLBiNet: A Cross-Sentence Collective Event Detection Network

The RNN-based encoder-decoder framework Cho et al. (2014); Sutskever et al. (2014); Bahdanau et al. (2015); Luong et al. (2015); Gu et al. (2016) consists of two components: a) an encoder which converts the source sentence into a fixed length vector $\mathbf{c}$ and b) a decoder is to unfold the context vector $\mathbf{c}$ into the target sentence. As is formalized in Gu et al. (2016), the source sentence $s_{i}$ is converted into a fixed length vector $\mathbf{c}$ by the encoder RNN,

$$\textbf{h}_{t}=f(\textbf{h}_{t-1},w_{t}),~{}\textbf{c}=\phi(\{\textbf{h}_{1},\dots,\textbf{h}_{n_{i}}\})$$

where $f$ is the RNN function, $\{\mathbf{h}_{t}\}$ are the RNN states, $w_{t}$ is the $t$-th token of source sentence, $\mathbf{c}$ is the so-called context vector, and $\phi$ summarizes the hidden states, e.g. choosing the last state $\mathbf{h}_{n_{i}}$. And the decoder RNN translates $\mathbf{c}$ into the target sentence according to:

$$\begin{gathered}\textbf{s}_{t}=f(y_{t-1},\textbf{s}_{t-1},\textbf{c})\\ p(y_{t}|y_{<t},s_{i})=g(y_{t-1},\textbf{s}_{t},\textbf{c})\end{gathered}$$

(1)

where $\mathbf{s}_{t}$ is the state at time $t$, $y_{t}$ is the predicted symbol at time $t$, $g$ is a classifier over the vocabulary, and $y_{<t}$ denotes the history $\{y_{1},\dots,y_{t-1}\}$.

Studies Bahdanau et al. (2015); Luong et al. (2015) have shown that summarizing the entire source sentence into a fixed length vector will limit the performance of the decoder. They introduced the attention mechanism to dynamically changing context vector $\mathbf{c}_{t}$ in the decoding process, where $\mathbf{c}_{t}$ can be uniformly expressed as

$$\displaystyle\mathbf{c}_{t}=\sum_{\tau=1}^{n_{i}}\alpha_{t\tau}\mathbf{h}_{\tau}$$

(2)

where $\alpha_{t\tau}$ is the contribution weight of $\tau$-th source token’s state to context vector at time $t$, $\mathbf{h}_{\tau}$ denotes the representation of $\tau$-th token.

We introduce the encoder-decoder framework to model ED task, mainly considering the following advantages: a) the separate encoder module is flexible in fusing sentence-level and document-level semantic information and b) the RNN decoder model (1) can capture sequential event tag dependency as the predicted tag vectors before $t$ will be used as input for predicting $t$-th symbol.

The encoder-decoder framework for ED task is slightly different from the general Seq2Seq task as follows: a) For ED task, the length of event tag sequence (target sequence) is known because its elements correspond one-to-one with tokens in the source sequence. However, the length of target sequence in the general Seq2Seq task is unknown. b) The vocabulary of decoder for ED task is a collection of event types, instead of words.

In this module, we encode the sentence-level contextual information for each token with Bidirectional LSTM (BiLSTM) and self-attention mechanism. Firstly, each token is transformed into comprehensive representation by concatenating its word embedding and NER type embedding. The word embedding matrix is pretrained by Skip-gram model Mikolov et al. (2013), and the NER type embedding matrix is randomly initialized and updated in the training process. For a given token $w_{t}$, its embedded vector is denoted as $\mathbf{e}_{t}$.

We apply the BiLSTM Zaremba and Sutskever (2014) model for sentence-level semantic encoding, which can effectively capture sequential and contextual information for each token. The BiLSTM architecture is composed of a forward LSTM and a backward LSTM, i.e., $\overrightarrow{\mathbf{h}}_{t}=\overrightarrow{\mbox{LSTM}}(\overrightarrow{\mathbf{h}}_{t-1},\mathbf{e}_{t}),~{}\overleftarrow{\mathbf{h}}_{t}=\overleftarrow{\mbox{LSTM}}(\overleftarrow{\mathbf{h}}_{t+1},\mathbf{e}_{t}).$ After encoding, the contextual representation of each token is $\mathbf{h}_{t}=[\overrightarrow{\mathbf{h}}_{t};\overleftarrow{\mathbf{h}}_{t}]$.

Attention mechanism between tokens within a sentence has been proven to further integrate long-range contextual semantic information. For each token $w_{t}$, its contextual representation is the weighted average of the semantic information of all tokens in the sentence. We apply the attention mechanism proposed by Luong et al. (2015) with the weights derived by

$$\begin{gathered}\alpha_{t,j}=\frac{\exp(z_{t,j})}{\sum_{m=1}^{n_{i}}\exp(z_{t,m})}\\ z_{t,m}=\tanh(\mathbf{h}_{t}^{\top}W_{sa}\mathbf{h}_{m}+b_{sa})\end{gathered}$$

(3)

And the contextual representation of $w_{t}$ is $\mathbf{h}^{a}_{t}=\sum_{j=1}^{n_{i}}\alpha_{t,j}\mathbf{h}_{j}$. By concatenating its lexical embedding and contextual representation, we get the final comprehensive semantic representation of $w_{t}$ as $\mathbf{x}_{t}=[\mathbf{h}^{a}_{t};\mathbf{e}_{t}]$.

The decoder layer for ED task is to generate a sequence of event tags corresponding to tokens. As is noted, the tag sequence (target sequence) elements and tokens (source sequence) are in one-to-one correspondence. Therefore, the context vector $\mathbf{c}$ shown in (1) and (2) can be personalized directly by $\mathbf{c}_{t}=\mathbf{x}_{t}$, which is equivalent to attention with degenerate weights. That is, $\alpha_{tt}=1$ and $\alpha_{t\tau}=0,~{}\forall\tau\neq t$.

In traditional Seq2Seq tasks, the target sequence length is unknown during the inference process, so only the forward decoder is feasible. However, for the ED task, the length of the target sequence is known when given source sequence. Thus, we devise a bidirectional decoder to model event inter-dependency within a sentence.

For current sentence $s_{i}$, the information we are concerned about can be summarized as recording which entities and tokens trigger which events. Thus, to summarize the information, we devise another LSTM layer (information aggregation module shown in Figure 1) with the event tag vector $\mathbf{y}_{t}$ as input. The information at $t$-th token is computed by

$$\displaystyle\tilde{\mathbf{I}}_{t}=\overrightarrow{\mbox{LSTM}}(\tilde{\mathbf{I}}_{t-1},\mathbf{y}_{t})$$

(6)

We choose the last state $\tilde{\mathbf{I}}_{n_{i}}$ as the summary information, which is $\mathbf{I}_{i}=\tilde{\mathbf{I}}_{n_{i}}$.

The sentence-level information aggregation module bridges the information across sentences, as the well-formalized information can be easily integrated into the decoding process of other sentences, enhancing the event-related signal.

In this module, we introduce a multiple bidirectional tagging layers stacking mechanism to aggregate information of adjacent sentences into the bidirectional decoder, and propagate information across sentences. The information $(\{\mathbf{y}_{t}\},\mathbf{I}_{i})$ obtained by the bidirectional decoder layer and information aggregation module has captured the event relevant information within a sentence. However, the cross-sentence information has not yet interacted. For a given sentence, as we can see in Table 1, its relevant information is mainly stored in several neighboring sentences, while distant sentences are rarely relevant. Thus, we propose to transmit the summarized sentence information $\mathbf{I}_{i}$ among adjacent sentences.

For the decoder framework shown in (4) and (5), the cross-sentence information can be integrated by extending the input with $\mathbf{I}_{i-1}$ and $\mathbf{I}_{i+1}$. Further, we introduce a multiple bidirectional tagging layers stacking mechanism shown in Figure 1 to iteratively aggregate information of adjacent sentences. The overall framework is named Multi-Layer Bidirectional Network (MLBiNet). As shown in Figure 1, a bidirectional tagging layer is composed of a bidirectional decoder and an information aggregation module. For sentence $s_{i}$, the outputs of $k$-th layer can be computed by

$$\begin{gathered}\overset{\rightarrow}{\mathbf{s}}_{t}=f_{\mbox{fw}}(\overset{\rightarrow}{\mathbf{y}}_{t-1}^{k},\overset{\rightarrow}{\mathbf{s}}_{t-1},\mathbf{x}_{t},\mathbf{I}_{i-1}^{k-1},\mathbf{I}_{i+1}^{k-1})\\ \overset{\leftarrow}{\mathbf{s}}_{t}=f_{\mbox{bw}}(\overset{\leftarrow}{\mathbf{y}}_{t+1}^{k},\overset{\leftarrow}{\mathbf{s}}_{t+1},\mathbf{x}_{t},\mathbf{I}_{i-1}^{k-1},\mathbf{I}_{i+1}^{k-1})\\ \overset{\rightarrow}{\mathbf{y}}_{t}^{k}=\tilde{f}(W_{y}\overset{\rightarrow}{\mathbf{s}}_{t}+b_{y})\\ \overset{\leftarrow}{\mathbf{y}}_{t}^{k}=\tilde{f}(W_{y}\overset{\leftarrow}{\mathbf{s}}_{t}+b_{y})\\ \mathbf{y}_{t}^{k}=[\overset{\rightarrow}{\mathbf{y}}_{t}^{k};\overset{\leftarrow}{\mathbf{y}}_{t}^{k}]\end{gathered}$$

(7)

where $\mathbf{I}_{i-1}^{k-1}$ is the sentence information of $s_{i-1}$ aggregated in (k-1)-th layer, and $\{\mathbf{y}_{t}^{k}\}$ are event tag vectors obtained in k-th layer. The equation suggests that for each token of source sentence $s_{i}$, the input of cross-sentence information is identical $[\mathbf{I}_{i-1}^{k-1},\mathbf{I}_{i+1}^{k-1}]$. It is reasonable as their cross-sentence information available is the same for each token of current sentence.

The iteration process shown in equation (7) is actually an evolutionary diffusion of the cross-sentence semantic and event information in the document. Specifically, in the first tagging layer, information of current sentence is effectively modeled by the bidirectional decoder and information aggregation module. In the second layer, information of adjacent sentences is propagated to current sentence by plugging in $\mathbf{I}_{i-1}^{1}$ and $\mathbf{I}_{i+1}^{1}$ to the decoder. In general, in the $k$-th ($k\geq 3$) layer, since $s_{i-1}$ has captured the information of sentence $s_{i-k+1}$ in the (k-1)-th layer, then $s_{i}$ can obtain information in $s_{i-k+1}$ by acquiring the information in $s_{i-1}$. Thus, as the number of decoder layers increases, the model will capture information from distant sentences. For $K$-layer bidirectional tagging model, the sentence information with the longest distance of K-1 can be captured.

We define the final event tag vector of $w_{t}$ as the weighted sum of $\{\mathbf{y}_{t}^{k}\}_{k}$ in different layers, i.e., $\mathbf{y}^{d}_{t}=\sum_{k=1}^{K}\alpha^{k-1}\mathbf{y}_{t}^{k},$ where $\alpha\in(0,1]$ is a weight decay parameter. It means that cross-sentence information can supplement to the current sentence, and the contribution gradually decreases as the distance increases when $\alpha<1$.

We note that the parameters of bidirectional decoder and information aggregation module at different layers can be shared, because they encode and propagate the same structured information. In this paper, we set the parameters of different layers to be the same.

In order to train the networks, we minimize the negative log-likelihood loss function $J(\theta)$,

$$J(\theta)=-\sum_{d\in D}\sum_{s\in d}\sum_{w_{t}\in s}\log p(O_{t}^{y_{t}}|d;\theta)$$

(8)

where $D$ denotes training documents set. The tag probability for token $w_{t}$ is computed by

$$\begin{gathered}O_{t}=W_{o}\mathbf{y}^{d}_{t}+b_{o}\\ p(O_{t}^{j}|d;\theta)=\exp(O^{j}_{t})/\sum_{m=1}^{M}\exp(O_{t}^{m})\end{gathered}$$

(9)

where $M$ is the number of event classes, $p(O_{t}^{j}|d;\theta)$ is the probability that assigning event type $j$ to token $w_{t}$ in document $d$ when parameter is $\theta$.

We performed extensive experimental studies on the ACE 2005 corpus to demonstrate the effectiveness of our method on ED task. It defines 33 types of events and an extra NONE type for the non-trigger tokens. We formalize it as a task to generate a sequence of 67-class event tag (with BIO tagging schema). The data splitting for training, validation and testing follows Ji and Grishman (2008); Chen et al. (2015); Liu et al. (2018); Chen et al. (2018); Huang and Ji (2020), where the training set contains 529 documents, the validation set contains 30 documents and the remaining 40 documents are used as testing set.

We evaluated the performance of three multi-layer settings with 1-, 2- and 3-layer MLBiNet, respectively. We use the Adam Kingma and Ba (2017) for optimization. In all three settings, we cut every 8 consecutive sentences into a new document and padding when needed. Each sentence is truncated or padded to make it 50 in length. We set the dimension of word embedding as 100, the dimension of golden NER type and subtype embedding as 20. We set the dropout rate as $0.5$ and penalty coefficient as $2*10^{-5}$ to avoid overfitting. The hidden size of semantic encoder layer and decoder layer is set to 100 and 200, respectively. The size of forward and backward event tag vectors is set to 100. And we set the batch size as 64, the learning rate as $5*10^{-4}$ with decay rate 0.99, the weight decay parameter $\alpha$ as $1.0$. The results we report are the average of $10$ trials.

For comparison, we investigated the performance of the following state-of-the-art methods: 1) DMCNN Chen et al. (2015), which extracts multiple events from one sentence with dynamic multi-pooling CNN; 2) HBTNGMA Chen et al. (2018), which models sentence event inter-dependency via a hierarchical tagging model; 3) JMEE Liu et al. (2018), which models the sentence-level event inter-dependency via a graph model of the sentence syntactic parsing graph; 4) DMBERT-Boot Wang et al. (2019), which augments the training data with external unlabeled data by adversarial mechanism; 5) MOGANED Yan et al. (2019), which uses graph convolution network with aggregative attention to explicitly model and aggregate multi-order syntactic representations; 6) SS-VQ-VAE Huang and Ji (2020), which learns to induct new event type by a semi-supervised vector quantized variational autoencoder framework, and fine-tunes with the pre-trained BERT-large model.

Table 2 presents the overall performance comparison between different methods with gold-standard entities. As shown, under 2-layer and 3-layer settings, our proposed model MLBiNet achieves better performance, surpassing the current state-of-the-art by $1.9$ points. More specifically, our models achieve higher recalls by at least $0.7$, $5.9$ and $5.2$ points, respectively.

The powerful encoder of BERT pre-trained model Devlin et al. (2018) has been proven to improve the performance of downstream NLP tasks. The 2-layer MLBiNet outperforms BERT-Boot (BERT-base) and SS-VQ-VAE (BERT-large) by $3.5$ and $1.9$ points, respectively. It proves the importance of event inter-dependency modeling and cross-sentence information integration for ED task.

When only information of current sentence is available, the 1-layer MLBiNet outperforms HBTNGMA by $2.9$ points. It proves that the hierarchical tagging mechanism adopted by HBTNGMA is not as effective as the bidirectional decoding mechanism we proposed. Intuitively, the bidirectional decoder models event inter-dependency explicitly by a forward decoder and a backward decoder, which is more efficient than hierarchies.

The existing event inter-dependency modeling methods Chen et al. (2015, 2018); Liu et al. (2018) aim to extract multiple events jointly within a sentence. To demonstrate that sentence-level event inter-dependency modeling benefits from cross-sentence information propagation, we evaluated the performance of our model in single event extraction (1/1) and multiple events joint extraction (1/n). 1/1 means one sentence that has one event; otherwise, 1/n is used.

The experimental results are presented in Table 3. As shown, we can verify the importance of cross-sentence information propagation mechanism and bidirectional decoder in sentence-level multiple events joint extraction based on the following results: a) When only the current sentence information is available, the 1-layer MLBiNet outperforms existing methods at least by $2.4$ points in 1/n case, which proves the effectiveness of bidirectional decoder we proposed; b) For ours 2-layer and 3-layer models, their performance in both 1/1 and 1/n cases surpasses the current methods by a large margin, which proves the importance of propagating information across sentences for single event and multiple events extraction. We conclude that it is the propagating information across sentences and bidirectional decoder which make cross-sentence joint event detection successful.

Table 4 presents the performance of the model in three decoder mechanisms: forward, backward and bidirectional decoder, as well as three multi-layer settings. We can reach the following conclusions: a) Under three decoder mechanisms, the performance of the proposed model will be significantly improved as the number of decoder layers increases; b) The bidirectional decoder dominates both forward decoder and backward decoder, and forward decoder dominates backward decoder; c) The information propagation across sentences will enhance event relevant signal regardless of the decoder mechanism applied. Among the three decoder models, the bidirectional decoder performs best because of its ability in capturing bidirectional event inter-dependency, which proves both the forward and backward decoders are critical for event inter-dependency modeling.

In information aggregation module, we introduce a LSTM shown in (6) to aggregate sentence information, and then propagate to other sentences via the bidirectional decoder. We compare other aggregation methods: a) concat means the sentence information is aggregated by simply concatenating the first and last event tag vector of the sentence, and b) average means the sentence information is aggregated by averaging the event tag vectors of tokens in the sentence. The experimental results are presented in Table 5.

Compared with the baseline 1-layer model, other three 2-layer settings equipped with information aggregation and cross-sentence propagation performs better. It proves that sentence information aggregation module can integrate some useful information and propagate it to other sentences through the decoder. On the other hand, the performance of LSTM and concat are comparable and stronger than average. Considering that the input of the information aggregation module is the event tag vector obtained by the bidirectional decoder, which has captured the sequential event information. Therefore, it is not surprising that LSTM does not have that great advantage over concat and average.