Large Language Models as Interpolated and Extrapolated Event Predictors

Event prediction, including tasks such as identifying key participants in an event or forecasting the occurrence of future events, has drawn significant attention in recent years across different domains (Zhao 2021; Onaolapo et al. 2022; Shi et al. 2024; Ye et al. 2024). The ability to successfully predict critical elements and accurately forecast human events is invaluable for proactive decision-making, risk management, and resource optimization. However, the complexity of underlying mechanisms and the inherent uncertainty of the future make event prediction a challenging endeavor (Wasserkrug 2009). These challenges necessitate advanced models that can effectively capture and interpret the dynamic patterns of events. In the past, the machine learning community has made significant progress in event reasoning and prediction by developing new methods based on recurrent neural networks (RNNs) and graph neural networks (GNNs). RNNs excel at capturing sequential information, while GNNs are adept at modeling relational knowledge within graphs (Pan et al. 2024; Ma et al. 2023). These networks have become foundational in the design of state-of-the-art (SOTA) event prediction frameworks. However, text information, such as news articles that describe events, provides rich semantic indicators and contextual backgrounds that are often underutilized. With the recent advancements in large language models (LLMs), we aim to explore their potential in understanding event contexts and inferring future events based on historical patterns and textual summaries. We identify several key challenges in leveraging LLMs for event prediction, from both interpolation and extrapolation perspectives (Wang et al. 2023; Shang and Huang 2024).

The high cost of closed-source LLMs: For interpolated object prediction (OP), we are asked to predict the missing object in a query quintuple, given the other four elements as well as historical knowledge tracing back to a certain sequence (Ma et al. 2023). Although nowadays generating several words looks simple (Ouyang et al. 2022), good prediction accuracy cannot be easily achieved, unless introducing SOTA commercialized LLMs and leveraging complicated prompting strategies (Ye et al. 2024), which consumes more tokens and leads to much more costly expenses in making frequent application programming interface (API) calls for OpenAI generative LLMs (OpenAI 2024b).

The constrained capability of open-source LLMs: For extrapolated multi-event forecasting (MEF), we are asked to forecast possible relation occurrences in a future time, given only historical knowledge tracing back to a certain time window without any query information (Deng, Rangwala, and Ning 2020). Compared with OP, MEF is more challenging for two reasons. First, MEF follows an extrapolation setting to predict future relations, without any auxiliary knowledge such as participating subjects, while the interpolation setting provides OP with the other four query elements, and the only thing left is to fill in the missing object blank. Second, MEF is on a daily basis while OP is on the quintuple level. Since there could be hundreds of quintuples per day (Boschee et al. 2015), the number of input tokens is much more demanding for LLMs involved in MEF to yield a valid downstream prediction than those in OP. However, the maximum number of input tokens for open-source LLMs is rather limited, such as 512 tokens for RoBERTa (Liu et al. 2019) and 1024 tokens for FLAN-T5 (Chung et al. 2024), as longer sequences would either be truncated by the tokenizer or jeopardize the overall performance (Kanakarajan and Sankarasubbu 2023).

To tackle these challenges, we propose LEAP, a unified framework that leverages large language models as event predictors. Considering both interpolation and extrapolation settings, we fine-tune an encoder-only LLM, RoBERTa (Liu et al. 2019), and an encoder-decoder LLM, FLAN-T5 (Chung et al. 2024), along with optimizing various customized downstream prediction heads, to achieve competitive accuracy in both OP and MEF. Our major contributions can be summarized as follows.

1. 1.

   We perform OP following two directions. First, as a ranking task, we leverage a fine-tuned encoder-only LLM, RoBERTa_BASE (Liu et al. 2019), to handle text summary in a query quintuple, and combine output embeddings back to a structural decoder through linear projection to make a prediction. Second, as a generative task following a question-answering (QA) format, where we design various prompt templates to build up questions and set correct objects as answers. We introduce an encoder-decoder LLM, FLAN-T5_BASE (Chung et al. 2024) for sequence-to-sequence generation, and leverage instruction fine-tuning to further enhance OP accuracy.

2. 2.

   We formulate MEF as a multi-label binary classification task. We design a simple yet effective prompt template for each event, and utilize a pre-trained RoBERTa_LARGE encoder  (Liu et al. 2019) to obtain quintuple-level embeddings. Subsequently, we customize a downstream head with self-attention (Vaswani et al. 2017) to perform weighted feature aggregation and predict possible relation occurrences in the future.

3. 3.

   We conduct comprehensive experiments with multiple sociopolitical ICEWS datasets (Boschee et al. 2015) involving different countries and various evaluation metrics, to analyze and demonstrate the validity and effectiveness of our approaches formulated in LEAP.

Event prediction has been comprehensively studied and various kinds of frameworks have been proposed to address specific challenges including, but not limited to prediction accuracy, optimization efficiency, and reasoning interpretability (Pan et al. 2024). Overall, these frameworks can be divided into three categories as follows (Liao et al. 2023).

Rule-based framework: The key components for rule-based frameworks are a list of pre-defined strategies to locate, identify, and extract parts of event quadruples or quintuples, which are assumed to be helpful for downstream inference (Liu et al. 2022). Usually, these strategies are ranked in numbers, and the constraints defined in strategies with larger numbers tend to be more relaxing than those with smaller ones to ensure the inclusion of all possible scenarios (Pan et al. 2023). Without any trainable parameters, rule-based frameworks are computationally efficient and easily interpretable by strictly following a list of rules. However, the implementation of manually defined rules is an exhaustive mining process within the event dataset itself, where the achieved prediction accuracy is not as competitive as those state-of-the-art (SOTA) methods with numerous parameters (Liao et al. 2023).

In-context learning framework: The key modules for in-context learning frameworks are generative large language models (LLMs) with frozen parameters (Dong et al. 2022). Various prompt templates can be designed to include few-shot learning examples and detailed instructions for completing different event prediction tasks (Lee et al. 2023). Although significant efforts for model training and downstream inference can be saved, there are two obvious shortcomings. First, for free-of-charge open-source LLMs, either they impose rather limited maximum input context length, such as 1024 tokens for BART (Lewis et al. 2019), 2048 tokens for LLaMA (Touvron et al. 2023a), and 4096 tokens for Llama 2 (Touvron et al. 2023b), or demand excessive memory space for local storage and inference, such as Llama 3.1_405B (Meta 2024), while the 8B and 70B versions are less powerful. Balancing the space-performance trade-off is always challenging for local implementations. Second, closed-source commercialized LLMs, GPT-4o (OpenAI 2024a) and Claude 3 (Anthropic 2024), are powerful generators which support much longer input context, but given abundant sociopolitical events (Boschee et al. 2015), the cost for making application programming interface (API) calls cannot be ignored (OpenAI 2024b).

Embedding-based framework: The key components for embedding-based frameworks are entity and relation embeddings updated by graph neural networks (GNNs) and recurrent neural networks (RNNs) (Ma et al. 2023). Extending quadruples to quintuples, handling additional text can be tricky. Specifically, for object prediction (OP), SeCoGD (Ma et al. 2023) applies the latent Dirichlet allocation algorithm (Blei, Ng, and Jordan 2003), which is pre-trained on collected and filtered text summaries, to separate quintuples into different context groups, and then leverages hypergraphs to collaborate intermediate embeddings among different groups. For multi-event forecasting (MEF), Glean (Deng, Rangwala, and Ning 2020) builds a word graph by calculating the point-wise mutual information (Church and Hanks 1990) between words tokenized and filtered from collected text summaries in an event dataset. Both SeCoGD and Glean are important benchmark methods, and by making appropriate use of encoding and generative LLMs, we aim to outperform these complex end-to-end frameworks, especially in terms of prediction accuracy.

Event interpolation refers to the retrieval or prediction of one or some missing facts at one specified timestamp (Cai et al. 2024), which corresponds to the definition of object prediction (OP) task, where we aim to predict a missing object $o_{q}$, given a query quintuple $(s_{q},r_{q},?,t_{q},x_{q})$ including known subject ($s_{q}$), relation ($r_{q}$), timestamp ($t_{q}$), and text summary ($x_{q}$), as well as selected historical knowledge tracing back to a pre-defined sequence length $l$, which can be structured as a list of graphs $\mathbf{G}_{\leq t_{q}}=\{G_{t_{q}-l},G_{t_{q}-l-1},...,G_{t_{q}-1},G_{t_{q}}\}$, or concatenated into textual corpus formatted as multiple strings $\mathbf{X}_{\leq t_{q}}$ (Ma et al. 2023; Liao et al. 2023).

Event extrapolation refers to the forecasting of events of interest in a future time, given only historical data (Cai et al. 2024), which complies with the definition of multi-event forecasting (MEF) task. In this task, we aim to forecast possible relation occurrences $r^{i}_{t_{n}}\in R$, where $i=1,2,...,\lvert R\rvert$ in the next timestamp $t_{n}$, given only historical information either in a structural format $\mathbf{G}_{<t_{n}}$ or a textual corpus format $\mathbf{X}_{<t_{n}}$ (Deng, Rangwala, and Ning 2020).

We start with a straightforward and simple prompt template $u_{i}$, as shown in Table 2, to express an event quintuples as plain language, which can be fed into an encoder-only LLM. The total number of simple prompts is equal to the total number of quintuples $Q$ for each dataset, where $i=1,2,...,\lvert Q\rvert$ (Boschee et al. 2015). Then, we leverage a pre-trained RoBERTa_LARGE, along with mean pooling $\text{Mean}(\cdot)$ for token-level aggregation, to generate an embedding vector $e_{i}$ for each quintuple prompt $u_{i}$, such that

where $e_{i}\in\mathbb{R}^{d_{h_{3}}}$ and $d_{h_{3}}$ is the output feature dimension for RoBERTa_LARGE encoder. The purpose of utilizing quintuple-level prompt engineering is to address the challenge of limited input context of open-source LLMs. Before feeding collected embeddings into downstream MEF layers, we need to convert them from the quintuple level back to the daily level with a purpose of incorporating as much history as possible for future forecasting. We write daily embeddings as $\mathbf{E}_{j}\in\mathbb{R}^{n_{j}\times d_{h_{3}}}$, where $j=1,2,...,\lvert T\rvert$, $T$ is the set of timestamps, and $n_{j}$ is the number of events on the $j$th day.

Based on Figure 3, tracing back to a pre-defined sequence length $l_{3}$, we feed concatenated embeddings $\mathbf{E}_{t_{n}-l_{3}\leq t<t_{n}}$ into a single-head self-attention layer (Vaswani et al. 2017), and then take a mean aggregation to collapse the dimension for multiple historical days, as we get

where $\tilde{\mathbf{E}}_{t_{n}}\in\mathbb{R}^{b\times d_{h_{3}}}$ indicates the aggregated embeddings and $b$ is the batch size for future days, $t_{n}=1,2,...,b$.

Subsequently, we introduce a fully-connected layer $z(\cdot)$ to adjust feature dimension and forecast possible relations in the future $t_{n}$, following an element-wise $\text{Sigmoid}(\cdot)$ with a threshold 0.5 to decide occurrences. We get

where $\mathbf{P}_{t_{n}}\in\mathbb{R}^{b\times\lvert R\rvert}$, and $R$ is the set of all unique relations. $z(\cdot)$ contains a trainable matrix $W_{z}\in\mathbb{R}^{d_{h_{3}}\times\lvert R\rvert}$ and vectorized bias $b_{z}\in\mathbb{R}^{\lvert R\rvert}$. Overall, for LEAP_MEF, the trainable modules are the self-attention layer and the fully-connected layer $z(\cdot)$, which are optimized with cross-entropy loss comparing predicted relation occurrences with true occurrences through back-propagation.

We conduct experiments on event datasets of three countries (Afghanistan, India, and Russia) from ICEWS which contains sociopolitical facts structured for crisis analysis, and each event is further enriched with several textual sentences sliced from authorized news articles (Boschee et al. 2015). Table 3 shows basic data statistics. The timestamps range from January 1, 2010, to February 22, 2016, and the minimum time gap is one day. All intermediate events from February 22, 2012 to January 1, 2013 are absent, so there are 1931 days in total, and the training/validation/test split is 0.8/0.2/0.2. All experiments involving module training and inference are implemented on a local server with $4\times 48$ GB NVIDIA L40S GPUs. For detailed experimental set up, such as hyperparameters and instruction fine-tuning results for RoBERTa_BASE encoder and FLAN-T5_BASE generator, we discuss them in the appendix.

In this paper, we propose LEAP to transform event prediction as language understanding and reasoning tasks. We investigated LLM-based frameworks for both interpolated tasks such as object prediction and extrapolated tasks such as multi-event forecasting. We apply quintuple-level prompt engineering to address the challenge of limited input context of open-source LLMs, and fine-tune RoBERTa_BASE as an encoder and FLAN-T5_BASE as a generator to achieve state-of-the-art event prediction accuracy on multiple real-world datasets. With fine-tuned open-source LLMs, we achieve better prediction accuracy than commercialized generative LLMs on sociopolitical events. Furthermore, we discuss the limitations of our approaches and explore several new ideas as potential future work.

The first limitation is that our OP and MEF approaches aim to simplify the design of existing event prediction frameworks (Ma et al. 2023; Deng, Rangwala, and Ning 2020) by introducing either encoding or generative LLMs, with enhanced prediction accuracy and affordable development being our major focus. As time evolves, the capability of pre-trained LLMs to reason the development of sociopolitical events can be further explored by integrating more advanced prompting strategies such as Chain-of-Thought (Wei et al. 2022) and ReAct (Yao et al. 2022).

The second limitation is that although multiple countries are involved, we only utilize ICEWS datasets (Boschee et al. 2015) for experiments, while other popular event datasets, such as GDELT (Leetaru and Schrodt 2013) and YAGO (Pellissier Tanon, Weikum, and Suchanek 2020), are not included, as our approaches are based on well-formulated event quintuples instead of quadruples which belong to the typical definition of temporal events (Cai et al. 2022). In other words, extending a quadruple to a quintuple by introducing a brief text summary can be regarded as a retrieval augmentation strategy (Lewis et al. 2020), which also demands more careful study.

One promising future direction is to evaluate the capability of LLMs in reasoning and understanding human event development. This involves meticulous design of experiments incorporating human feedback. Another important direction is to enhance LLMs’ temporal and spatiotemporal understanding of the world by integrating retrieval-augmented generation and knowledge graphs.

In this section, we describe detailed hyperparameters for all sub-tasks of LEAP, and provide additional experimental results on LLM fine-tuning. Our experiments are implemented on a Ubuntu 22.04 server with 512GB RAM ($8\times 64\text{GB DDR5}$) and $4\times 48\text{GB}$ NVIDIA L40S GPUs.