Zero- and Few-Shot Event Detection via Prompt-Based Meta Learning

Event detection refers to the task of classifying event types upon input text. While event detection has achieved progress under the supervised training paradigm Ji and Grishman (2008); Lin et al. (2020); Wadden et al. (2019); Liu et al. (2020); Du and Cardie (2020); Lu et al. (2021); Liu et al. (2022), zero- and few-shot classification remains a challenge due to the lack of prior knowledge and annotated examples. For the zero-shot setting, relevant literature focuses on predefined event knowledge or heuristics to classify unseen events Huang et al. (2018); Huang and Ji (2020); Lyu et al. (2021); Zhang et al. (2021b); Yu et al. (2022); Yu and Ji (2023); Zhan et al. (2023). Similarly, zero-shot contrastive learning requires unlabeled examples in training to learn class-separating features Zhang et al. (2022b). Under the few-shot setting, prototypical networks and prompt-based tuning improve detection performance via prototype matching and alignment to language pretraining Deng et al. (2020); Cong et al. (2021); Schick and Schütze (2021); Lai et al. (2021); Li et al. (2022b). Overall, existing approaches focus on either zero- or few-shot event detection without considering a unified framework for both settings. Moreover, current zero-shot methods require additional resources for training, making such methods less realistic in real-world event detection. As such, we propose a meta learning framework MetaEvent for both zero- and few-shot event detection, where neither prior knowledge nor unlabeled examples are required to detect unseen events.

Prompt learning uses a predefined template with slots (i.e., A <mask> event) to instruct language models on the desired task, where the slot prediction is used to derive the final output Brown et al. (2020); Liu et al. (2021). By leveraging the pretraining objective and designing prompt templates, pretrained large language models can be adapted for zero- or few-shot downstream tasks Houlsby et al. (2019); Raffel et al. (2020). Soft and multi-task prompts further improve the zero- and few-shot performance of language models on unseen tasks Lester et al. (2021); Sanh et al. (2021). Cloze-based prompts are proposed for the event detection task, where the predicted output can be mapped to the event types using verbalizer or mapping heuristics Schick and Schütze (2021); Li et al. (2022b); Zhang et al. (2022b). Nevertheless, previous prompt methods adopt the inefficient two-step paradigm (i.e., trigger identification and classification) and are not designed for both zero- and few-shot event detection. Therefore, we integrate both steps with a trigger-aware soft verbalizer for efficient forward passing in meta training. Moreover, we consider both zero- and few-shot scenarios with our prompt-based meta learning framework MetaEvent. By leveraging the proposed components, our approach demonstrates considerable improvements compared to existing methods.

Meta learning (i.e., learning to learn) has demonstrated superiority in few-shot learning Finn et al. (2017); Nichol et al. (2018); Rajeswaran et al. (2019). Existing methods learn class-wise features or prototypes in the metric space for quick adaptation to few-shot tasks Vinyals et al. (2016); Snell et al. (2017); Sung et al. (2018). Model-agnostic meta learning (MAML) leverages the second-order optimization to find the optimal initial parameters for a new task Finn et al. (2017). Approximation methods of second-order MAML demonstrates comparable performance while requiring significantly reduced computational resources Finn et al. (2017); Nichol et al. (2018); Rajeswaran et al. (2019). Meta learning has also been applied to tasks like online learning, domain adaptation and multi-task learning Finn et al. (2019); Li and Hospedales (2020); Wang et al. (2021a). For event detection, meta learning is proposed to improve the performance on small-size data via memory-based prototypical networks and external knowledge Deng et al. (2020); Shen et al. (2021).

To the best of our knowledge, meta learning-based methods for both zero- and few-shot event detection is not studied in current literature. However, detecting unseen and rare events is necessary in many applications. An example can be detecting emergency events on social media, where both unseen and rare events can be present (e.g., pandemic, safety alert etc.). Therefore, we propose MetaEvent: a meta learning framework for zero- and few-shot event detection. MetaEvent exploits seen events via trigger-aware prompting and a carefully designed meta objective, and thereby improving the zero- and few-shot performance with class-separating and generalizable features.

To efficiently perform prompt-based meta training for event detection, we design a one-step model that integrates the trigger identification and classification stages. This is because a two-step approach (e.g., P4E Li et al. (2022b)) requires extensive computational resources to obtain the gradients in the inner- and outer-loop optimization in MetaEvent. Consequently, we design an efficient model (as illustrated in Figure 2) that integrates attentive trigger features to avoid additional forward passes.

Different from existing trigger-aware methods Ding et al. (2019), the proposed model innovatively uses both attentive trigger features and prompt output to predict the event types. The attentive trigger features $\bm{t}$ can be computed using an integrated trigger classifier and attention weights from the pretrained language model. Specifically, a trigger classifier is trained upon each token features to perform binary classification (i.e., whether input token is trigger token). In inference, the classifier predicts the probabilities $\bm{p}$ of input tokens being classified as trigger words. To better estimate the importance of predicted trigger tokens, we design an attentive reweighting strategy to select informative trigger features from the input context. The idea behind our attentive reweighting strategy is to leverage attention scores from the model to select more relevant features. The attention scores reveal different importance weights of the context tokens and thus, can be used to compute ‘soft’ trigger features based on the semantics. Formally, our attentive reweighting strategy computes weights $\bm{w}\in\bm{R}^{L_{c}}$ using trigger probabilities $\bm{p}$ and attention scores $A\in\bm{R}^{H\times L_{c}\times L_{c}}$ of the context span from the last transformer layer. The weight of the $i$-th token $w_{i}$ in $\bm{w}$ is computed via

$$w_{i}=\sigma\bigg{(}\bm{p}\odot\frac{1}{H}\sum_{j}^{H}\bigg{(}\sum_{k}^{L_{c}}A_{j,k}\bigg{)}\bigg{)}_{i},$$

(2)

where $H$ is the number of attention heads, $L_{c}$ is the context length, $\odot$ and $\sigma$ denote elementwise product and softmax functions. Base on input $\bm{x}_{c}$ and $\bm{w}$, the attentive trigger features $\bm{t}$ can be computed as the weighted sum of token features, i.e.,

$$\bm{t}=\sum_{i=1}^{L_{c}}w_{i}\bm{f}_{\bm{\theta}}(\bm{x}_{c})_{i}.$$

(3)

For the event classification, we design a prompt-based paradigm using a predefined prompt and a trigger-aware soft verbalizer. Specifically, we preprocess the input context by prepending the prompt ‘A <mask> event’ to transform the prediction into a masked language modeling (MLM) task. The pretrained encoder model and MLM head fill the <mask> position with a probability distribution $\bm{v}$ over all tokens. Then, our trigger-aware soft verbalizer maps the predicted distribution $\bm{v}$ to an output event type. Unlike predefined verbalizer functions Li et al. (2022b), we design a learnable verbalizer based on MLM predictions $\bm{v}$ and attentive trigger features $\bm{t}$. For the $N$-way few-shot setting, the trigger-aware soft verbalizer with weights $\bm{W}\in\bm{R}^{(|\bm{v}|+|\bm{t}|)\times N}$ and bias $\bm{b}\in\bm{R}^{N}$ predicts the output label with GELU activation via

$$\hat{y}=\arg\max(\mathrm{GELU}([\bm{v};\bm{t}])\bm{W}+\bm{b}).$$

(4)

For zero-shot event detection, we use the concatenated features $[\bm{v};\bm{t}]$ to project input to unseen event types via the Hungarian algorithm.

Provided with the training and evaluation sets from sampled tasks, we present the formulation of our MetaEvent and our methods for the zero- and few-shot training. The designed framework leverages meta training to search for optimal parameters $\bm{\theta}$. Once trained, the event detection model quickly adapts to unseen tasks even without examples Finn et al. (2017); Yue et al. (2023).

Given a set of tasks $\{\bm{\mathcal{T}}_{i}\}_{i=1}^{M}$ and model $\bm{f}$ parameterized by $\bm{\theta}$, MetaEvent aims at minimizing the overall evaluation loss of the tasks (as in Equation 1). MetaEvent consists of an inner-loop optimization stage (i.e., $\mathcal{A}lg$) and an outer-loop optimization staget that minimizes the overall loss w.r.t. $\bm{\theta}$. For the inner-loop update, $\mathcal{A}lg$ denotes gradient descent with learning rate $\alpha$, i.e.:

$$\mathcal{A}lg(\bm{\theta},\bm{\mathcal{D}}^{\mathrm{train}})=\bm{\theta}-\alpha\nabla_{\bm{\theta}}\mathcal{L}(\bm{\theta},\bm{\mathcal{D}}^{\mathrm{train}})=\bm{\phi},$$

(5)

we denote the updated parameter set with $\bm{\phi}$. In the outer-level optimization, we are interested in learning an optimal set $\bm{\theta}$ that minimizes the meta loss on the evaluation sets. The learning is achieved by differentiating through the inner-loop optimization (i.e., $\mathcal{A}lg$) back to the initial parameter set $\bm{\theta}$, which requires the computation of second-order gradients or first-order approximation (as shown in Figure 2). Specifically, we derive the gradients w.r.t. $\bm{\theta}$:

$$\frac{d\mathcal{L}}{d\bm{\theta}}=\frac{d\bm{\phi}}{d\bm{\theta}}\nabla_{\bm{\phi}}\mathcal{L}(\mathcal{A}lg(\bm{\theta},\bm{\mathcal{D}}^{\mathrm{train}}),\bm{\mathcal{D}}^{\mathrm{test}}),$$

(6)

notice that $\mathcal{A}lg(\bm{\theta},\bm{\mathcal{D}}^{\mathrm{train}})$ is equivalent to $\bm{\phi}$. Component $\nabla_{\bm{\phi}}\mathcal{L}(\mathcal{A}lg(\bm{\theta},\bm{\mathcal{D}}^{\mathrm{train}}),\bm{\mathcal{D}}^{\mathrm{test}})$ refers to first-order gradients w.r.t. the task-specific parameter set $\bm{\phi}$ (i.e., $\mathcal{L}\rightarrow\bm{\phi}$). The left component $\frac{d\bm{\phi}}{d\bm{\theta}}$ tracks parameter-to-parameter changes from $\bm{\phi}$ to $\bm{\theta}$ through $\mathcal{A}lg$ (i.e., $\bm{\phi}\rightarrow\bm{\theta}$), which involves the computation of the Hessian matrix. As the estimation of the matrix $\frac{d\bm{\phi}}{d\bm{\theta}}$ requires extensive computational resources, we provide both first-order and second-order implementations for MetaEvent.

We now introduce our training objective $\mathcal{L}$ for MetaEvent. Based on the proposed model in Section 4.1, our loss function contains two classification losses: trigger classification loss $\mathcal{L}_{\mathrm{trigger}}$ and event classification loss $\mathcal{L}_{\mathrm{event}}$ (i.e., negative log likelihood loss). To enlarge the inter-class event discrepancy for improved zero- and few-shot performance, we additionally propose a contrastive loss $\mathcal{L}_{\mathrm{con.}}$ based on the maximum mean discrepancy (MMD) Gretton et al. (2012); Yue et al. (2021, 2022b, 2022a). In particular, we measure the discrepancy between two different event types by estimating the MMD distance. MMD computes the distance between two event distributions using an arbitrary number of input features drawn from these event types. Mathematically, MMD distance between input features $\bm{X}$ and $\bm{Y}$ can be computed as:

	$\displaystyle\mathcal{D}(\bm{X},\bm{Y})$	$\displaystyle=\frac{1}{|\bm{X}||\bm{X}|}\sum_{i=1}^{|\bm{X}|}\sum_{j=1}^{|\bm{X}|}k(\psi(\bm{x}^{(i)}),\psi(\bm{x}^{(j)}))$		(8)
		$\displaystyle+\frac{1}{|\bm{Y}||\bm{Y}|}\sum_{i=1}^{|\bm{Y}|}\sum_{j=1}^{|\bm{Y}|}k(\psi(\bm{y}^{(i)}),\psi(\bm{y}^{(j)}))$
		$\displaystyle-\frac{2}{|\bm{X}||\bm{Y}|}\sum_{i=1}^{|\bm{X}|}\sum_{j=1}^{|\bm{Y}|}k(\psi(\bm{x}^{(i)}),\psi(\bm{y}^{(j)})),$

where $k$ represents the Gaussian kernel and $\psi$ represents the feature mapping function defined by the transformer network (i.e., feature encoder).

Based on the MMD distance, we further compute the inter-class distances for all pairs of event types. Suppose $\bm{X}_{i}$ represents the set of trigger-aware event features (i.e., concatenation of $[\bm{v};\bm{t}]$ as in Section 4.1) of the $i$-th class, the contrastive loss can be formulated for the $N$-way setting as:

$$\mathcal{L}_{\mathrm{con.}}=-\frac{1}{N(N-1)}\sum_{i=1}^{N}\sum_{j=1,j\neq i}^{N}\mathcal{D}(\bm{X}_{i},\bm{X}_{j}),$$

(9)

in which we compute $N\times(N-1)$ pairs of inter-class distances, such inter-class distances are maximized (by taking the negative values) in the meta objective function to encourage class-separating features in MetaEvent, and therefore improves both zero- and few-shot performance in event detection.

The overall framework of MetaEvent is presented in Figure 2. The right subfigure illustrates the proposed model that integrates attentive trigger features for prompt-based event detection. The left subfigure illustrates the meta training paradigm of MetaEvent, where the initial parameter $\bm{\theta}$ is updated and evaluated upon sampled tasks using the proposed contrastive loss in Equation 10. Then the gradients w.r.t. $\bm{\theta}$ can be computed (via second-order optimization or first-order approximation) to update the initial model. Unlike previous works Schick and Schütze (2021); Cong et al. (2021); Li et al. (2022b); Zhang et al. (2022b), we design a trigger-aware model for efficient training and inference on event detection tasks. Moreover, we propose a meta learning framework MetaEvent with a fine-grained contrastive objective function for zero- and few-shot event detection, which encourages generalizable and class-separating features across both seen and unseen event types.

We first report zero-shot results on all datasets in Table 1, which is divided into two parts by the used datasets. We perform $10$-way event detection on unseen tasks from the disjoint test sets and report the evaluation results, the best results are marked bold, the second best results are underlined. We observe: (1) the zero-shot performance on FewEvent is comparatively higher than MAVEN for both baselines and MetaEvent, possibly due to the increased coverage of event domains in MAVEN. (2) MetaEvent performs the best on both datasets, in particular, MetaEvent achieves $35.9\%$ average improvements on F1 compared to the second best-performing method. (3) Despite lower performance on MAVEN, MetaEvent outperforms all baseline methods with up to $50.8\%$ improvements on F1, suggesting the effectiveness of the proposed meta training algorithm in detecting unseen events.

We further study the effectiveness of the proposed contrastive loss quantitatively in zero-shot experiments by varying scaling factor $\lambda_{c}$. Specifically, we choose $\lambda_{c}$ from $0$ to $5$ and evaluate the performance changes on both datasets. The results are visually presented in Figure 3, from which we observe: (1) by applying the contrastive loss (i.e., $\lambda_{c}\neq 0$) in the meta objective, the zero-shot performance consistently improves regardless of the choice of $\lambda_{c}$; (2) despite huge improvements, carefully chosen $\lambda_{c}$ is required for the best zero-shot performance (up to $\sim 100\%$ increases across all metrics). Overall, the contrastive objective proposed in MetaEvent is particularly effective in learning class-separating features, and thereby improves the zero-shot event detection performance.

We additionally present qualitative analysis of our zero-shot results by comparing the feature visualization (via T-SNE) with and without the proposed contrastive loss. We use color to represent different event types and present the visualization in Figure 4. With the contrastive loss (right subfigure), the model generates class-separating features on unseen events compared to MetaEvent trained without contrastive loss (left subfigure). Examples of the same event type are also more aggregated, showing improved clustering results, which can be combined with trigger predictions for identifying unseen event types. In sum, the proposed contrastive loss demonstrates effectiveness in zero-shot settings and consistently outperforms baselines.

To examine the effectiveness of our method for both zero- and few-shot scenarios, we also perform few-shot experiments on both datasets. The $5$-shot and $10$-shot event detection experiment results are presented in Table 2, where all evaluation event types are used ($10$-way for FewEvent and $45$-way for MAVEN³³3Similar to Chen et al. (2021), we perform binary classification for each of the event types in MAVEN as multiple event labels may exist on the same input context.). We observe: (1) all baseline methods and MetaEvent achieve improved performance in few-shot event detection compared to the zero-shot results. For example, MetaEvent achieves $36.3\%$ F1 improvement in $5$-shot setting on FewEvent. (2) For MAVEN, the average performance of all baseline methods are comparatively lower than FewEvent, indicating the challenge of few-shot classification with increased number of event types. (3) MetaEvent achieves the best results, outperforming the second best method in F1 by $3.0\%$ (on FewEvent) and $38.1\%$ (on MAVEN) on average across both few-shot settings. Overall, MetaEvent achieves state-of-the-art performance in event detection even only with few examples.

We now perform ablation studies in the few-shot setting to evaluate the effectiveness of the proposed component in MetaEvent. In particular, we remove the proposed attentive trigger features (i.e., trigger), trigger-aware soft verbalizer (i.e., verbalizer) and outer-loop optimization (i.e., meta learner) in sequence to observe the performance changes on FewEvent. The results are reported in Table 3. For all components, we observe performance drops when removed from MetaEvent. In the $5$-shot setting, F1 score reduces by $1.6\%$ and $12.9\%$ respetively when removing the attentive trigger features and trigger-aware soft verbalizer. The results suggest that proposed components are effective for improving few-shot event detection.

Finally, we study the influence of prompt designs and report the results on FewEvent in Table 4. In particular, we select from prompt A: ‘A <mask> event’, B: ‘This text describes a <mask> event’, C: ‘This topic is about <mask>’ and D: ‘[Event: <mask>]’. From the results we observe: for $5$-shot event detection, prompt A and B perform the best while prompt B and D achieves better performance with $0.9644$ and $0.9592$ F1 in $10$-shot setting. On average, prompt B outperforms all other prompt designs in the F1 metric, indicating that well-designed instructions may slightly improve few-shot event detection results.