Bi-Directional Iterative Prompt-Tuning for Event Argument Extraction

Early deep learning methods use various neural networks to capture the dependencies in between event triggers and event arguments to extract event arguments, such as convolutional neural network (CNN)-based models (Chen et al., 2015), recurrent neural network (RNN)-based models (Nguyen et al., 2016; Sha et al., 2018) and graph neural networks (GNN)-based models (Liu et al., 2018; Dai et al., 2021). As pre-trained language models have been proven to be powerful in language understanding and generation (Devlin et al., 2019; Liu et al., 2019; Lewis et al., 2020), some PLM-based methods have been proposed to extract event arguments. These methods can be divided into two categories: fine-tuning and prompt-tuning ones.

Fine-tuning methods aim to design a variety of neural network models to transfer pre-trained language models to EAE task. According to the modeling manner of EAE task, existing fine-tuning work can be further divided into three groups: classification-based methods (Wang et al., 2019; Wadden et al., 2019; Lin et al., 2020; Ma et al., 2020; Xiangyu et al., 2021); machine reading comprehension-based methods (Du and Cardie, 2020; Li et al., 2020; Liu et al., 2020); generation-based methods (Paolini et al., 2020; Lu et al., 2021). Prompt-tuning methods aim to design a template to provide useful prompt information for pre-trained language models to extract event arguments (Li et al., 2021; Ma et al., 2022; Hsu et al., 2022; Liu et al., 2022). For example, Li et al. (2021) create a template for each event type based on the event ontology definition and model the EAE task as the conditional text generation. This method acquires event arguments by comparing the designed template with the generated natural language text. Hsu et al. (2022) improve the method of Li et al. (2021) by replacing the non-semantic placeholder tokens in the designed template with words with role label semantics.

The core of prompt-tuning is to transform the given downstream task into a form that is consistent with a training task of the pre-trained language models (Liu et al., 2021). As prompt-tuning makes better use of prior knowledge contained in pre-trained language models, this new paradigm is beginning to become popular in NLP tasks and has achieved promising performance (Seoh et al., 2021; Han et al., 2021; Cui et al., 2021; Hou et al., 2022; Hu et al., 2022; Chen et al., 2022). For example, Cui et al. (2021) use candidate entity spans and entity type label words to obtain templates, and recognize entities based on the pre-trained generative language model’s score for each template. Hu et al. (2022) convert the text classification task to a masked language modeling problem by predicting the word filled in the "[MASK]" token, and propose a knowledgeable verbalizer to map the predicted word into a label. Chen et al. (2022) consider the relation extraction problem as a cloze task and use the relation label semantic knowledge to initialize the virtual label word embedding for each relation label.

As the most common ACE dataset provides entity mention, entity type and entity coreference information, we use these entity information to formalize event argument extraction into the argument role prediction problem of entities. The detailed problem description is as follow: Given a sentence $S$, an event trigger $t$ with event type, and $n$ entities $\{e_{1},e_{2},...,e_{n}\}$, the goal is to predict the argument role of each entity in the event triggered by $t$ and output a set of argument roles $\{r_{1},r_{2},...,r_{n}\}$.

In this paper, the argument role prediction problem is casted as a cloze-style task through a template $T(\cdot)$ and verbalizer. For the trigger $t$ and entity $e_{i}$, a template $T(t,e_{i},[\texttt{MASK}])$ is constructed to query the argument role that the entity $e_{i}$ plays in the event triggered by $t$. For example in Figure 1(b), the template $T(fired,Iraqis,[\texttt{MASK}])$ can be set as "For the attack event triggered by the fired, the person, Iraqis, is [MASK]", where "attack" represents the event type of the trigger "fired" and "person" represents the entity type of the entity "Iraqis". Then the input of the $i$-th entity $e_{i}$ is:

$$x_{i}=S\ [\texttt{SEP}]\ T(t,e_{i},[\texttt{MASK}]).$$

(1)

The verbalizer is a mapping from the label word space to the argument role space. Let $l_{j}$ denote the label word that is mapped into the role $r_{j}$, the confidence score that the $i$-th entity is classified as the $j$-th role type is:

$$s_{ij}=C_{i}([\texttt{MASK}]=l_{j}),$$

(2)

where $C_{i}$ is the output of a pre-trained masked language model at the masked position in $x_{i}$, i.e. the confidence score of each word in the dictionary filled in the [MASK] token.

Figure 2 presents the overall architecture of our bi-directional iterative prompt-tuning method, consisting of a forward iterative prompt (FIP) and a backward iterative prompt (BIP). The forward iterative prompt predicts the argument role of each entity iteratively from left to right until argument roles of all entities are obtained. For example in Figure 2, the order of entities is "$Iraqis\rightarrow missiles\rightarrow AAA\rightarrow aircraft$".

In order to utilize the predicted argument role information to classify the current entity into the correct role, we introduce the argument roles of the first $i$-$1$ entities into the template of the $i$-th entity. The template of the $i$-th entity in the forward iterative prompt can be represented as:

$$FIP(e_{i})=T(t,e_{1},\overrightarrow{l_{1}},...,e_{i-1},\overrightarrow{l_{i-1}},e_{i},[\texttt{MASK}]),$$

(3)

where $\overrightarrow{l_{j}}$ is the role label word of the $j$-th entity predicted by the forward iterative prompt. For example in Figure 2, $\overrightarrow{l_{1}}$ is the word "attacker". Then the confidence score distribution of the $i$-th entity over all argument roles in the forward iterative prompt can be computed by

$$\overrightarrow{\mathbf{s}_{i}}=MLM(S\ [\texttt{SEP}]\ FIP(e_{i})).$$

(4)

$\overrightarrow{l_{i}}$ is the word corresponding to the argument role with the highest value in $\overrightarrow{\mathbf{s}_{i}}$.

Similarly, the backward iterative prompt predicts the argument role of each entity in a right-to-left manner. The argument role confidence score distribution of the $i$-th entity in the backward iterative prompt can be computed by:

	$$\displaystyle BIP(e_{i})=T(t,e_{n},\overleftarrow{l_{n}},...,e_{i+1},\overleftarrow{l_{i+1}},e_{i},[\texttt{MASK}]),$$		(5)
	$$\displaystyle\overleftarrow{\mathbf{s}_{i}}=MLM(S\ [\texttt{SEP}]\ BIP(e_{i})).$$		(6)

Then we can obtain the final argument role confidence score distribution of the $i$-th entity by

$$\mathbf{s}_{i}=\overrightarrow{\mathbf{s}_{i}}+\overleftarrow{\mathbf{s}_{i}}.$$

(7)

Finally, the argument role label with the highest score is chosen as the role prediction result.

To tackle the problem that an argument role may have different definitions in different types of events, we reconstruct the set of argument role types and design a semantical verbalizer. Specifically, we further divide the argument role that participates in multiple types of events into multiple argument roles that are specific to event types. For example, the "Entity" role is divided into "Elect:Entity", "Meet:Entity", and etc. Since the "Place" role has the same meaning in all types of events, we do not consider to divide it.

For each new argument role, the semantical verbalizer constructs a virtual word to represent the role and initializes the representation of the virtual word with the semantic of the argument role. Let a $m$-word sequence $\{q_{i1},q_{i2},...,q_{i,m}\}$ denote the semantic description of the argument role $r_{i}$, the initial representation of the label word $l_{i}$ that is mapped into the role $r_{i}$ can be computed by:

$$\mathbf{E}(l_{i})=\frac{1}{m}\sum\limits^{m}_{j=1}\mathbf{E}(q_{ij}),$$

(8)

where $\mathbf{E}$ is the word embedding table of a pre-trained masked language model.

For redefined argument roles, different argument roles may have the same semantics, such as "Appeal:Adjudicator" and "Sentence:Adjudicator". Therefore, it is easy to misclassify the entity with "Appeal:Adjudicator" role into the "Sentence:Adjudicator" role. In order to solve the problem, we use the event structure information to extract arguments. For an event with the "Appeal" type, its role label can only be "Appeal:Defendant", "Appeal:Adjudicator" and "Appeal:Plaintiff".

To take full advantage of event type, trigger, and entity information, the designed template should contain event types, triggers, entity types, and entity mentions. Since some entity types and event types are not human-understandable words, such as "PER" and "Phone-Write", we need to convert each entity (event) type into a human-understandable text span. For example, we use "person" and "’written or telephone communication" as the text spans for "PER" and "Phone-Write" respectively.

Let $M_{i}=\{\varepsilon_{i1},\varepsilon_{i2},...,\varepsilon_{id}\}$ denote the entity mention set of the $i$-th entity, the word sequence of the $i$-th entity can be represented as:

$$\hat{e}_{i}=\varepsilon_{i1}\ or\ \varepsilon_{i2}\ or\ ...\ or\ \varepsilon_{id}.$$

(9)

We use $w^{t}$ to denote the text span of event type of the given trigger and $w_{i}^{e}$ to denote the text span of the entity type of the $i$-th entity. For the given trigger $t$ and $i$-th entity $e_{i}$, three different templates of forward iterative prompt are designed as follows:

• •

  Hard Template: All known information are connected manually with natural language. "For the $w^{t}$ event triggered by the $t$, the $w_{1}^{e}$, $\hat{e}_{1}$, is $\overrightarrow{l_{1}}$, … , the $w_{i-1}^{e}$, $\hat{e}_{i-1}$, is $\overrightarrow{l_{i-1}}$, the $w_{i}^{e}$, $\hat{e}_{i}$, is [MASK]"

• •

  Soft Template: Add a sequence of learnable pseudo tokens after all known information. "$w^{t}$ $t$ $w_{1}^{e}$ $\hat{e}_{1}$ $\overrightarrow{l_{1}}$ … $w_{i-1}^{e}$ $\hat{e}_{i-1}$ $\overrightarrow{l_{i-1}}$ $w_{i}^{e}$ $\hat{e}_{i}$ [V1] [V2] [V3] [MASK] [V4] [V5] [V6]"

• •

  Hard-Soft Template: All known information are connected with learnable pseudo tokens. "[V1] $w^{t}$ [V2] $t$ [V3] [V4] $w_{1}^{e}$ [V5] $\hat{e}_{1}$ [V6] $\overrightarrow{l_{1}}$, … , [V4] $w_{i-1}^{e}$ [V5] $\hat{e}_{i-1}$ [V6] $\overrightarrow{l_{i-1}}$ [V4] $w_{i}^{e}$ [V5] $\hat{e}_{i}$ [V6] [MASK]"

Pseudo tokens are represented by "[Vi]". The embedding of each pseudo token is randomly initialized and optimized during training.

During training, gold argument roles are used to generate the template of each entity in forward iterative prompt and backward iterative prompt. The optimization objective is to ensure that the masked language model can predict argument role accurately in both forward iterative prompt and backward iterative prompt. We use $\overrightarrow{p_{t,i}}$ and $\overleftarrow{p_{t,i}}$ to represent the probability of the entity $e_{i}$ playing each role type in the event triggered by $t$ in forward and backward iterative prompt respectively. The loss function is defined as follows:

	$$\displaystyle\overrightarrow{p_{t,i}}=softmax(\overrightarrow{\mathbf{s}_{i}}),\ \ \overleftarrow{p_{t,i}}=softmax(\overleftarrow{\mathbf{s}_{i}}),$$
	$$\displaystyle\mathbb{L}=-\sum\limits_{t\in\mathbb{T}}\sum\limits^{n_{t}}_{i=1}(\log(\overrightarrow{p_{t,i}}(\tilde{r}_{t,i}))+\log(\overleftarrow{p_{t,i}}(\tilde{r}_{t,i}))),$$		(10)

where $\mathbb{T}$ is the event trigger set in the training set, $n_{t}$ is the number of entities contained in the same sentence as the event trigger $t$, and $\tilde{r}_{t,i}$ is the correct argument role of the $i$-th entity playing in the event triggered by $t$.

We evaluate our proposed method on the most widely used event extraction dataset, ACE 2005 English dataset¹¹1https://catalog.ldc.upenn.edu/LDC2006T06 (Doddington et al., 2004). Following the previous work (Wadden et al., 2019; Lin et al., 2020; Ma et al., 2022), the dataset is pre-processed and divided into training/development/test set, where $33$ event subtypes, $7$ entity types and $22$ argument roles are considered in the processed dataset. As event argument extraction task is only focused on, we use gold entities and event triggers to conduct experiments.

We use Bert-base(containing around 110 millions parameters) (Devlin et al., 2019) and Roberta-base(containing around 125 millions parameters) (Liu et al., 2019) models to predict the masked words and train each model with AdamW, where the batch size is set to $4$ and the learning rate is set to $1e$-$5$. For the low-resource setting, we generate some subsets containing $(1\%,5\%,10\%,20\%,50\%,75\%)$ of the fulling training set in the same way as  (Hsu et al., 2022). In each experiment, the masked language model is trained by a subset and evaluated by the fulling development and test sets. All experiments are run on a NVIDIA Quadro P4000 GPU.

Two categories of state-of-the-art methods are compared with our proposed method.

Since we use an entity as a unit for argument role prediction, an event argument is correctly identified if the entity corresponding to the argument is predicted to be the non-None role type. The argument is further be correctly classified if the predicted role type is the same as the gold label.

For the above baselines, they consider that an event argument is correctly classified only if its offsets and role type match the golden argument, which can be called "strict match (SM)". In order to compare our model with baselines more fairly, we use a "flexible match (FM)" method to evaluate these baselines, that is, an argument is correctly classified if its offsets match any of the entity mentions co-referenced with the golden argument and role type match the golden argument.

Same as the previous work, the standard micro-averaged Precision(P), Recall(R), and F1-score(F1) are used to evaluate all methods.

Table 1 compares the overall results between our model and baselines, from which we have several observations and discussions.

(1) BIP(Roberta) gains the significant improvement in event argument extraction. The F1-scores of BIP(Roberta) are more than $9\%$ higher than those of all baselines obtained by the strict match evaluation method. Even using the flexible match method to evaluate baselines, the BIP(Roberta) method also outperforms the state-of-the-art ONEIE(Roberta) by $3.24\%$ increase of F1-score in term of argument identification and $3.53\%$ increase of F1-score in term of role classification.

(2) Comparing with the strict match, the flexible match achieves $5\%$ to $7\%$ F1-score improvements in term of argument identification and role classification. These results indicate that the trained argument extraction models can indeed identify the entity mention co-referenced with the golden argument as the event argument. In addition, in the actual application scenarios, we only pay attention to which entity is the event argument, not which mention in an entity is the event argument. Therefore, it is more reasonable and efficient to predict argument roles in unit of entity than entity mention.

(3) Roberta-version methods outperform Bert-version methods. In particular, for our proposed BIP method, Roberta further gains $3.77\%$ and $4.8\%$ F1-score improvements on argument identification task and role classification task respectively. These improvements can be explained by Roberta using a much larger training dataset than Bert and removing the next sentence prediction task. In the following experiments, we only consider Roberta-version methods.

Table 2 presents an ablation study of our proposed BIP method. BIP(forward) only considers the forward iterative prompt to extract event arguments. BIP(backward) only considers the backward iterative prompt. BIP-BI does not use a bi-directional iterative strategy to consider argument interactions, i.e. predicts the argument role of each entity separately. BIP-SV replaces our designed semantical verbalizer with a human-written verbalizer, where each label word is manually selected from a pre-trained language model vocabulary. BIP-BI-SV uses neither the bi-directional iterative strategy nor the semantical verbalizer. Some observations on the ablation study are as follows:

(1) Compared with the method BIP, the performance of BIP(forward) and BIP(backward) is decreased by $1.56\%$ and $2.76\%$ F1-score in term of role classification, respectively. These results clearly demonstrate that the bi-directional iterative prompt-tuning can further improve the performance by comparing with one direction.

(2) Comparing with the methods BIP-BI and BIP-BI-SV, the methods BIP and BIP-SV can further improve the performance of role classification in terms of $1.65\%$ and $0.14\%$ increase of F1-score, respectively. These results suggest that the bi-directional iterative strategy is useful for event argument extraction. In addition, we notice that the improvement brought by our bi-directional iterative strategy for the method BIP-BI is higher than BIP-BI-SV. This suggests that the more accurate the independent predicted argument role of each entity, the greater improvement the bi-directional iterative strategy will bring to the performance of argument extraction.

(2) The methods BIP and BIP-BI are respectively outperform the methods BIP-SV and BIP-BI-SV by $2.64\%$ and $1.13\%$ F1-score in term of role classification. These results illustrate that our semantical verbalizer is more effective than a human-written verbalizer for event argument extraction.

Figure 4 presents the performance of our BIP, BIP-BI and two state-of-the-art methods in both low-resource and high-resource data scenarios. We can observe that the variation of F1-score has a trend of rising with the increase of the training data. Comparing the fine-tuning method ONEIE, prompt-tuning methods BIP, BIP-BI and PAIE obviously improve the performance of role classification in low-resource data scenarios. This result shows that prompt-tuning methods can more effectively utilize the rich knowledge in PLMs than fine-tuning methods.

Even using flexible match to evaluate the prompt-tuning method PAIE, our method BIP and BIP-BI achieve better performance in both low-resource and high-resource data scenarios. The main reason is that our method can make use of the entity information and predicted argument roles when constructing the template. We notice that the performance of BIP is worse than that of BIP-BI, when the ratio of training data is less than $20\%$. This is because when the number of training data is too small, the probability of argument roles being correctly predicted is low. If the bi-directional iterative strategy is adopted, the wrongly predicted argument roles will be used for template construction, which will further degrade the performance of EAE.

In order to showcase the effectiveness of our method BIP, we sample three sentences from the ACE 2005 English test dataset to compare the event argument extraction results by BIP, BIP(forward), BIP(backward), BIP-BI and BIP-SV methods.

In Sentence 1 of Table 3, the method without the bi-directional iterative strategy BIP-BI can only identify the entity "Bush" as the "Entity" role. For the entity "Putin", the methods with the forward iterative prompt BIP, BIP(forward), BIP-SV can correctly classify it into the "Entity" role. This attributes to that the information that entity "Bush" is the "Entity" argument is introduced into the template construction of the entity "Putin". We also notice that "Bush" and "Putin" are both misclassified in the BIP(backward) method, where the error role information of "Putin" is passed to the classification of "Bush". In addition, for the entity "[he, Erdogan]" in Sentence 2, the method only with the forward iterative prompt BIP(forward) misclassifies the entity "[Tomahawk, missiles]" into the "Instrument" role. These results show that the argument roles of contextual entities can provide useful information for the role identification of the current entity. However, only considering argument interactions in one direction may degrade the performance of event argument extraction.

In Sentence 3, the method BIP-SV misclassifies the entity "SEC" into the "Entity" role. For the human-written verbalizer of BIP-SV, the word "judge" is selected as the label word of role "Adjudicator". It is difficult to associate the entity "SEC" with the word "judge". In the semantical verbalizer, we use the text sequence "the entity doing the fining" to describe the semantic of "Adjudicator" role in the "Fine" event. Since the pre-trained language models can easily identify the entity "SEC" as "the entity doing the fining", the methods with semantical verbalizer can correctly identify the entity "SEC" as the "Adjudicator" role. The result verifies the effectiveness of our designed semantical verbalizer.

In this section, we compare three different templates introduced in Section 3.4 to investigate how different types of templates affect the performance of EAE. For the BIP-BI method, the performances of hard template, soft template and hard-soft template are comparable. Since the hard-soft template combines the manual knowledge and learnable virtual tokens, it achieves the best performance. However, the hard-soft template performs worst for the BIP method. Unlike the BIP-BI method which only considers event trigger and current entity information, BIP introduces the predicted argument role information into the template. Therefore, there are so many learnable pseudo tokens in the hard-soft template, resulting in poor performance.