Large Language Models as Event Forecasters

To make good use of the contextualized potential embedded within the fifth element, the text summary briefly describing a TKG quadruple, we collect and concatenate summaries from all quintuples to form a large textual corpus for the training/validation/test split respectively. Subsequently, we leverage the corpus to fine-tune an encoder-only LLM, RoBERTa-base, with the standard masked language modeling loss (Liu et al. 2019). The RoBERTa-base encoder takes a textual summary $x$ as input, and outputs an embedding vector of $\boldsymbol{e}\in\mathbb{R}^{50265}$ for each token, then we apply mean pooling to obtain the sentence embedding $\boldsymbol{e_{x}}\in\mathbb{R}^{50265}$ for the textual summary of each quintuple.

Before utilizing the sentence embeddings, we evaluate the effects of masked language modeling fine-tuning (Liu et al. 2019) and leverage perplexity (Jelinek et al. 1977) as the evaluation metric. Based on Table 1, we can observe a significant decrease in test perplexities for all three datasets after fine-tuning. Therefore, we expect that better semantic meanings can be encoded into the sentence embeddings obtained from the fine-tuned RoBERTa-base, which has stronger contextualized understanding abilities.

Going back to structural frameworks, the architectures of our approach and other competitors are briefly summarized in Figure 1 and Table 2, where the GNN module is relational graph convolutional network (RGCN) (Schlichtkrull et al. 2018) and the RNN module is gated recurrent unit (GRU) (Cho et al. 2014). For the structural decoder which combines queries with embeddings from GNN and RNN and eventually ranks all candidate objects after Softmax activation, we refer to ConvTransE (Shang et al. 2019). Compared with SeCoGD (Ma et al. 2023), where the context IDs are acquired from pre-trained clustering algorithms (Blei, Ng, and Jordan 2003), we believe that the semantic information encoded in those queries’ sentence embeddings, along with linear projection and non-linear ReLU activation, can significantly enrich the encoder optimization and improve the prediction accuracy.

When framing object prediction as a ranking task, we use hits at 1/3/10 as evaluation metrics, to rank the correct object among all candidates based on output probabilities after Softmax activation. For this ranking task, we consider both the original triples and the reversed triples by swapping the subject and the object, which follows the same setting as SeCoGD and can be regarded as a data augmentation strategy. Accordingly, the number of candidates becomes the number of unique entities in each dataset.

Based on Table 3, we can observe that our approach with sentence embeddings encoded from fine-tuned RoBERTa can achieve much better hits at 1/3/10 scores than other competitors. In addition, the overall scores will slightly increase if we trace back to more historical days. However, the historical sequence length, as a hyperparameter, cannot grow unlimitedly due to increasing computational cost.

We have demonstrated the impressive capabilities of a fine-tuned encoder-only LLM to support a structural framework in the embedding space. Our next question is will LLMs be able to predict objects directly as a generative predictor? In other words, if we stack historical knowledge, queries, and instructions into a textual prompt $C_{in}$, and feed $C_{in}$ into a generative LLM, such as FLAN-T5 (Chung et al. 2024) or GPT-3.5 (OpenAI 2023), how likely can it generate the missing object $o_{q}$ correctly? To answer this question, based on Figure 2, we frame object prediction as a generative task, where we introduce ROUGE scores (Lin 2004) to directly compare the true object and the predicted object as two strings, instead of ranking all candidate objects. Therefore, to transfer from the ranking task to the generative task, we directly read out the first candidate which has the highest output probability as the predicted object, iteratively across all test samples, and then compute the ROUGE scores for selected structural frameworks. Based on Table 5, we can observe that our structural approach with sentence embeddings can still outperform SeCoGD (Ma et al. 2023) significantly, following similar trends as in Table 3.

We try to construct the generative predictor from two perspectives. On the one hand, we can prompt commercialized LLMs, such as GPT-3.5-Turbo-Instruct (GPT-3.5 for short) (OpenAI 2023), to directly generate predictions. However, there are several obvious shortcomings of this approach. First, prompting GPT-3.5 is not free of charge, and GPT-4 is much more expensive than GPT-3.5. Second, Chain-of-Thought prompting hinders post-processing and automatic evaluation given too many queries. Third, due to the closed-source property of GPT-3.5, we can do nothing other than few-shot learning. On the other hand, we can fine-tune an open-source LLM, such as FLAN-T5 (Chung et al. 2024), an encoder-decoder LLM to generate predictions. Since FLAN-T5 has much fewer parameters and a shorter input/output token length than GPT-3.5, it may have much weaker commonsense generative capabilities compared with GPT-3.5. Therefore, we fine-tune a FLAN-T5 model for each dataset specifically following a standard question-answering format, where the “question” is a 5-shot learning prompt with one query along with some explanations and instructions, and the “answer” is the object entity.

For this generative task, we consider only the original triples without swapping the subject and object, which is different from the data augmentation setting in previous ranking task. This is due to the rather limited maximum input context length of FLAN-T5, which makes it challenging to stack more reversed triples into the prompt as additional examples and queries. To address the different setting scenario, when reading out the first candidate for structural frameworks, we only read out from original triples and do not consider those reversed triples. Therefore, after transferring from the ranking task to the generative task, all methods can share the same setting before computing ROUGE scores to make sure a fair comparison.

As shown in Table 4, we fine-tune FLAN-T5 with a 5-shot learning prompt (original prompt). To evaluate the generalizability, we remove the 5 learning examples to build a zero-shot prompt. To further evaluate the effectiveness introduced by text, we design another 5-shot learning prompt (no-text prompt) but remove all text summaries in 5 learning examples and the query as an ablation study.

Based on Table 5, we can observe that fine-tuning FLAN-T5 as a generative predictor can yield much better ROUGE scores even than our best structural approach with sentence embeddings. In addition, the zero-shot prompt can achieve competitively similar scores as the original prompt, demonstrating good generalization. However, the no-text prompt has significantly degraded scores across all three datasets, which indicates that the fifth element of a query quintuple, textual summary, is very important to enhance the contextualized understanding abilities of a generative LLM during instruction fine-tuning.

Overall, we can see that the fine-tuned FLAN-T5 is a great generative predictor. However, what about directly prompting GPT-3.5, can it make better use of text summaries? To answer this question, as well as to save computational cost in terms of making frequent OpenAI API (OpenAI 2023) calls to predict large amount of missing objects, we select the first 5000 test samples, and prompt GPT-3.5 only with the original prompt. Similar to Table 5, we collect the outputs iteratively and compute the ROUGE scores based on those true objects without any additional post-processing.

Based on Table 6, we can observe that, given the original prompt, GPT-3.5 has much worse generative performance than the fine-tuned FLAN-T5. GPT-3.5 also has lower ROUGE scores than our structural approach with sentence embeddings. Therefore, simply prompting commercialized closed-source LLMs is not always the best option, especially for one or several specifically formatted problems, where relatively smaller open-source LLMs can be fine-tuned to obtain improved performance. For object prediction, the reasoning is rather intuitive, in that there could be thousands of unique objects in one dataset totally (Boschee et al. 2015), and the open-source FLAN-T5 can learn what most objects should be during fine-tuning, while the closed-source GPT-3.5 could only derive the output based on its pre-trained commonsense knowledge and the provided prompt, which is impossible to fit all object candidates as the maximum number of input tokens for GPT-3.5-Turbo-Instruct is 4096 (OpenAI 2023).

Since multi-event forecasting operates on a daily level and each day can include hundreds of quintuples, resulting in concatenated strings that far exceed thousands of tokens, substantial TKG sampling is required. Specifically, we aim to collect only a small subset of quintuples from each day, typically ranging from tens to twenties, to use as historical knowledge in the prompt. However, after various sampling strategies, we find that even with Longformer (Beltagy, Peters, and Cohan 2020), which is based on RoBERTa (Liu et al. 2019) and fine-tuned for a much longer maximum input token length (4096 tokens instead of 512 tokens for RoBERTa), daily prompts constructed by sampling TKG quintuples within the same day would yield unsatisfactory results compared with Glean’s random initialization without optimization (Deng, Rangwala, and Ning 2020). Therefore, we aim to develop an approach where we do not need to pay the price of sampling while handling TKG structures and historical sequences without introducing GNNs and RNNs.

We start with a straightforward simple prompt template, as shown in Table 7, to incorporate the structure of a quintuple. The total number of prompts is equal to the total number of quintuples for each ICEWS dataset (Boschee et al. 2015). Then, we leverage the encoder-only LLM, RoBERTa-large, along with mean pooling, to generate an embedding vector $\boldsymbol{e_{p}}\in\mathbb{R}^{50265}$ for each quintuple prompt. Now, since each prompt contains only one quintuple, the maximum input length is no longer challenging, and we do not need to perform significant sampling. Given this simple prompt template, we hope that the RoBERTa-large encoder could easily grasp the structural as well as the contextualized meanings embedded within each quintuple.

Subsequently, as shown in Figure 3, these quintuple embeddings tracing back to several historical days will be fed into a downstream prediction head, which contains two modules for completing the multi-event forecasting task.

The first module is a single-head self-attention layer (Vaswani et al. 2017). The self-attention mechanism is applied towards different quintuples within the same historical day, and our motivation is to grasp potentially useful information by identifying and weighing the importance difference across all quintuples within each historical day. Otherwise, it would be simply a mean aggregation to collapse the dimension of multiple quintuples per day.

The second module is simply a fully-connected layer, where the output dimension is equal to the unique number of relations, followed by element-wise Sigmoid activation with a threshold of 0.5 to decide occurrences.

During experiments, we only optimize the downstream prediction head, while freezing the LLM encoder. Table 8 shows data statistics of five countries from ICEWS (Boschee et al. 2015). Each country has 1931 days for training, 224 days for validation, and 224 days for test. Table 9 briefly summarizes our approaches and other competitors, such as the deep neural network (DNN) (Bengio et al. 2009), the dynamic GCN (Deng, Rangwala, and Ning 2019), the temporal GCN (Zhao et al. 2019), the recurrent event network (RENET) (Jin et al. 2019), and Glean (Deng, Rangwala, and Ning 2020) with composition-based multi-relational GCN (CompGCN) (Vashishth et al. 2019). Compared with other SOTA methods, our approaches neither involve GNNs nor RNNs, which significantly saves the design efforts on updating intermediate embeddings.

Based on Table 10, with good prompt engineering and appropriate downstream prediction head design, we can achieve better multi-event forecasting accuracy than a carefully structured framework, such as Glean (Deng, Rangwala, and Ning 2020), which is equipped with delicate GNNs for TKGs and RNNs for historical sequences. We also conduct ablation studies by removing the self-attention (SA) mechanism (Vaswani et al. 2017). According to Table 10, we can also observe that introducing self-attention before daily aggregation, which collapses the dimension of multiple historical days, is important. Intuitively, we believe there involves more complicated relationship among daily quintuples that continuously affect each other with time going by, which requires more detailed research as future work.

Last but not least, before optimizing the prediction head, we also try to fine-tune RoBERTa-large with masked language modeling loss (Liu et al. 2019), by stacking all quintuple prompts to form a large textual corpus. Our motivation is to make the RoBERTa encoder more compatible with such a knowledge-intensive task. However, comparing the fine-tuned LLM with the pre-trained version, we find that the accuracy improvement is rather trivial. We provide two perspectives to explain such scenario:

• •

  Optimization: We train our downstream prediction head thoroughly, providing a small learning rate and many epochs while saving model checkpoints based on the best validation recall. Though the prompt embeddings as inputs are different before and after LLM fine-tuning, both layers in the downstream prediction head could arrive at the same local optima after thorough optimization.

• •

  Problem Formulation: The pre-trained RoBERTa-large encoder has already been very powerful, especially given relatively short prompts compared to its maximum input length. Furthermore, the multi-event forecasting task is to predict possible relation occurrences in the future given only historical knowledge. This setting itself has much uncertainty. Recall that the perplexity is used to evaluate how well a language model has learned the distribution of the text on which it has been trained (Jelinek et al. 1977). Therefore, having a high perplexity within the originally pre-trained RoBERTa-large may not accurately reflect the performance of multi-event forecasting.

First, all of our approaches for object prediction and multi-event forecasting aim to simplify the design of existing TKG learning frameworks by introducing either encoder-only or encoder-decoder LLMs. As time evolves, the capability of pre-trained LLMs to reason and infer the development of human societies can be further explored.

Second, although multiple countries are involved, we only utilize ICEWS datasets (Boschee et al. 2015) for experiments, while other popular TKG datasets, such as GDELT (Leetaru and Schrodt 2013) and YAGO (Pellissier Tanon, Weikum, and Suchanek 2020), are not considered.

Third, a typical TKG unit should be a quadruple, instead of a quintuple (Cai et al. 2022). In other words, extending a quadruple to a quintuple by introducing a brief text summary can be regarded as a retrieval augmentation strategy (Lewis et al. 2020), which also demands more careful study.

The first future direction is to transfer from quintuples back to quadruples for better generalization. Although we may not start with abundant text summaries which are important for LLMs’ contextualized understanding, many existing TKG learning frameworks, either rule-based or embedding-based, hold a default setting that TKGs are built up with quadruples instead of quintuples (Liao et al. 2023). If it is necessary to introduce the fifth element as helpful text, then we aim to develop our own retrieval augmentation methods to slice informative sentences from open-source knowledge base, such as Wikidata (Vrandečić and Krötzsch 2014).

Second, we aim to propose a new problem formulation for TKG learning. While object prediction is relatively simple and straightforward, multi-event forecasting is highly challenging due to the need to predict the occurrences of hundreds of unique relations simultaneously for a future day. Therefore, we aim to find a balance between these tasks. One potential future direction is to closely examine each unique relation on each future day and dynamically adjust the prompt context. Although this approach may be computationally expensive, it promises to provide more detailed historical knowledge during optimization.