Improving Zero-Shot Event Extraction via Sentence Simplification

With the proliferation of social media, microblogs and online news, we are able to gain a real-time understanding of events happening around the world. By ingesting large unstructured datasets and converting them into structured formats such as (actor, event, target) tuples we can make rapid progress in systems for event forecasting  (Ramakrishnan et al., 2014), real-time event coding  (Saraf and Ramakrishnan, 2016) or other applications that can grant organizations a strategic advantage. Historically, this has been enabled by efforts such as ICEWS¹¹1https://dataverse.harvard.edu/dataverse/icews & GDELT²²2https://www.gdeltproject.org/. These systems rely on event extraction technology to populate their knowledge bases. Fig. 1 gives an example of an event ‘Bring lawsuit against’ from the ICEWS dataset. Extraction involves identifying entities (businessman, employees) corresponding to argument roles ‘Actor’ and ‘Target’. The event is triggered by the predicate ‘sued’ in the figure. However, the extraction technology employed by these

systems relies on pattern-based approaches that use handcrafted patterns designed to extract entities and events (Boschee et al., 2013). Even though pattern-based methods have high precision, they fail to work on unseen event types and with new event categories. Hence, there is a need to explore extraction methods that can extend beyond a fixed domain. Modern approaches for event extraction (Chen et al., 2015; Nguyen et al., 2016; Wadden et al., 2019) rely on fine-grained annotations and suffer from data scarcity issues and error propagation due to pipeline systems.

With the success of large scale pretrained language models on machine reading comprehension (MRC) tasks (Devlin et al., 2019a; Liu et al., 2019; Huang et al., 2018), a new paradigm for event extraction based on MRC has surfaced (Du and Cardie, 2020; Liu et al., 2020). In this approach, event argument extraction is posed as a span extraction problem from a context conditioned on a question for each argument. This approach is promising because it mitigates some of the issues faced by traditional approaches, such as relying on upstream systems to extract entities/triggers and hence sidestepping the error propagation problem in pipeline systems. It also gives rise to the possibility of zero-shot event extraction and hence the ability to extend to new domains which is traditionally hard due to difficulties in collecting high-quality labeled training data. However, MRC models struggle with long-range dependencies and syntactic complexities. For instance, Liu et al. (2020) observe that one typical error from their MRC-based extraction system is related to long-range dependency between an argument and a trigger, accounting for 23.4% errors on the ACE-2005 event dataset Doddington et al. (2004) (here “long-range” denotes that the distance between a trigger and an argument is greater than or equal to 10 words). Du and Cardie (2020) observe that one of the failure modes of their extraction system is sentences with complex sentence structures containing multiple clauses, each with trigger and arguments. These observations make a promising case for complexity reduction or context simplification for MRC systems.

To mitigate the above problem and to reduce the syntactic complexity we propose an unsupervised approach that is guided by a scoring function that incorporates syntactic fluency, simplicity and the confidence of an MRC model(§ 2) Our key contributions are:

1. 1.

   The exploration of sentence and context simplification to help mitigate thelong-range dependency problem for MRC based zero-shot event extraction (§ 4).

2. 2.

   Experimental results on the ICEWS political event datasets including a detailed analysis of areas of selective superiority of our approach.

3. 3.

   Followup analysis to demonstrate how simplification can be controlled based on desired factors such as preference for high performance for a certain argument role.

Reading comprehension models can be brittle to subtle changes in context. They can be thrown-off by syntactic complexity, especially when the questions are not specific and do not include words overlapping with the context. Moreover, long range dependencies between the trigger/predicate and the argument can also throw the model off. For this purpose, we propose an MRC-guided Unsupervised Sentence Simplification algorithm (RUSS), that iteratively performs deletions and extractions from the context in search for a higher-scoring candidate. The score function incorporates components that ensure sentence fluency, information preservation and the confidence of the target MRC model. Fig. 2 gives an overview of the proposed approach.

Concretely, given that an event has been detected in a sentence, the task is to identify the arguments of the detected event. For instance, in Fig. 1 the task is to identify the arguments ‘Actor’ and ‘Target’ of the event ‘Bring lawsuit against’. The algorithm takes the QA pairs corresponding to each argument role as input. The QA generation procedure for the dataset used in this paper for evaluation is outlined in Appendix B. Table 1 shows the generated QA-pair for the arguments Actor and Target for the event shown in Fig. 1.

We evaluate RUSS on the ICEWS event dataset³³3https://dataverse.harvard.edu/dataverse/icews from years 2013 to 2015. In this dataset, event data consists of coded interactions between socio-political actors (i.e., cooperative or hostile actions between individuals, groups, sectors and nation states) mapped to the CAMEO ⁴⁴4https://parusanalytics.com/eventdata/data.dir/cameo.html ontology. These events are in the form of triples consisting of a source actor, an event type (according to the CAMEO taxonomy of events), and a target actor. For evaluation, we aim to extract the Actor and Target roles for event types shown in Table  5. We use the data from years 2013-2015 for training and 2016 for testing/evaluation in section 4.4. See Appendix for distribution of event types.

The results of zero-shot extraction on the ICEWS dataset are outlined in Table  2. In the baselines used, simplification is performed with score function exponents for $\nu_{lm}$ as $a=1.5$ and $\nu_{entity}$ as $b=1$ held constant while varying $c$ for $\nu_{pred}$, $r_{1}$ for $\nu_{actor}$ and $r_{2}$ for $\nu_{target}$. With no simplification we get F1 scores of 0.412 and 0.354 for actor and target roles respectively. For the most basic setting for simplification with $c=0$, $r_{1}=1$ and $r_{2}=1$ scores improve by 4.6% for actor prediction to 0.431 and by 10.4% to 0.391 for target prediction respectively which shows that simplifying context can further improve a powerful model like BERT in a cross-domain zero-shot setting. For actor prediction, out of 37,894 records we find that for 10.99% records, F1 score improves after simplification, for 6.54% records F1 decreased after simplification and for the rest the score remained unchanged. For target prediction, for 17.4% records scores improve where as for 7.9% records the scores decreased and for the rest of the records, the scores remained unchanged. After introducing the predicate score ($c=1$) we see that these improvements drop slightly. This is counter-intuitive, because one would expect model performance to improve when relevant predicates are present in the context. We attribute this behavior to the MRC model leveraging the language priors in the training data to predict the answers. For instance, the model could predict the subject of the predicate as an answer for ‘Who’ type of questions.

Next, we increase the coefficients of Actor and Target roles from 1 to 3. The reason why we choose an odd number for this exponent is because sometimes for bad candidates the RC scores can be negative and since all the scores are combined in a multiplicative way, raising a negative score to an even power would reverse the desired effect. Observing the results in rows 5 & 6 of Table 2 we can see that percentage of sentences with similar scores before and after simplification have increased. We can also observe that percentage of sentences for which scores decrease after simplification have also decreased. We can conclude that by raising the coefficients of role specific scores we can make the simplification models more robust to inaccurate simplifications for those roles. We also observe, when $r_{1}=3$, we get the highest F1 for actor prediction, an improvement of 5.6% over no simplification and for $r_{2}=3$ we can an F1 on-par with the highest obtained in row 2. Our results clearly indicate the benefit of simplification over no simplification and also the gradual improvement in scores when the argument coefficients $r_{1},r_{2}$ are varied from 0 to 3.

Event extraction(EE) has been an active area of research in the past decade. In EE, supervised approaches usually rely on manually labeled training datasets and handcrafted ontologies.  Li et al. (2013) utilize the annotated arguments and specific keyword triggers in text to develop an extractor. Supervised approaches have also been studied using dependency parsing by analyzing the event-argument relations and discourse of event interactions (McClosky et al., 2011). These approaches are usually limited by the availability of the fine-grained labeled data and required elaborately designed features. Recent work formulates event argument extraction as an MRC task. A major challenge with this approach is generating a dataset of QA pairs.  Liu et al. (2020) propose a method combining template based and unsupervised machine translation for question generation.  Du and Cardie (2020) follow a template approach and show that more natural the constructed questions better the event extraction performance. However, none of these methods directly aim to address the long-range dependency problem using simplification.

Automatic text simplification (ATS) systems aim to transform original texts into their lexically and syntactically simpler variants. The motivation for building the first ATS systems was to improve the performance of machine translation systems and other text processing tasks, e.g. parsing, information retrieval, and summarization  Chandrasekar et al. (1996). In the context of extraction, Zhang et. al.  Zhang et al. (2018) show that pruning dependency trees to remove irrelevant structures can improve relation extraction performance. Efforts have been made to incorporate syntactic dependencies into models in an effort to mitigate this problem 2016; 2018; 2020. Recently,  Mehta et al. (2020) have used sentence simplification as a preprocessing step for improving machine translation. Edit-based simplification has been investigated to a great degree to improve the readability of the text (Kumar et al., 2020; Dong et al., 2019; Alva-Manchego et al., 2017). To the best of our knowledge this is the first work that studies sentence simplification for improving MRC-based event extraction.

In this work, we motivated the need for MRC-based event extraction paradigm especially for zero-shot scenarios(§ 1). Next, we discussed the long-range dependency problem ubiquitously faced by event extraction systems. We proposed a context simplification algorithm to reduce the syntactic complexity of the context aided by MRC-system feedback to address the problem(§ 2). Our results indicate that simplification can not only aid MRC systems in a zero-shot setting(§ 4.1- 4.3) but also when they’re finetuned on in-domain data(§ 4.4).

Although, the proposed method can be useful to boost extraction performance when offline computations can be afforded, it may be difficult to scale in real-time use cases. Reasons include – 1) RC system inference time while running the algorithm; 2) call latency if using APIs and 3) search algorithms can be computationally expensive. A promising way to improve efficiency would be to generate parallel training data for simplification using the RUSS method offline and guide the MRC model during training using the generated data. This can be done by using attention masks over the deleted words obtained using RUSE. We leave this approach for future work.