Zero is Not Hero Yet: Benchmarking Zero-Shot Performance of LLMs for Financial Tasks¹¹1This working paper is an ongoing research project, and feedback is greatly appreciated.

Understanding or deciphering the monetary policy stance of the central banks is an important NLP task to understand financial markets better. Several studies (Rozkrut et al., 2007; Tobback et al., 2017; Hansen et al., 2018; Cieslak et al., 2019; Tsukioka and Yamasaki, 2020; Bennani et al., 2020; Shah et al., 2023a) have found that the words used by central banks have an impact on the market, although the extent of this impact varies depending on the communication style and chair. Recent work by Hansen and Kazinnik (2023) uses ChatGPT to decode the FOMC post-meeting statements. In order to understand the capabilities of LLMs in decoding central bank communications, we use open-source labeled data that combines meeting minutes, press conference transcripts, and speeches (Shah et al., 2023a). In this dataset, each sentence is labeled one of the three (hawkish, dovish, and neutral) labels which we use as a sequence classification task.

Sentiment analysis is a popular task in the financial domain as it correlates with the market sentiment which influences price movements. In the finance literature, the sentiment dictionary developed by Loughran and McDonald (2011) is used as bag-of-words extensively for sentiment analysis. Garcia et al. (2023) refine the dictionary for bag-of-words and claim it to be SOTA in the finance literature. On the other side computational linguistics literature has grown exponentially over the last decade. After the introduction of sentence-level financial sentiment data by Malo et al. (2014), many models have been trained for sentiment analysis tasks. Soon after the release of BERT (Devlin et al., 2018), FinBERT (Araci, 2019) was developed for financial sentiment analysis. To understand how well generative LLMs can do on this important task, we use Financial Phrasebank sentiment analysis data⁷⁷7https://huggingface.co/datasets/financial_phrasebank developed by Malo et al. (2014) for the financial sentiment classification task. The dataset contains multiple versions. Here we use the data where there is 100% annotation agreement. We perform three class (positive, negative, and neutral) sequence classification task.

Extraction of numerical claims from the financial text like analysts’ reports, earnings calls, news, etc is helpful in forecasting volatility of the stock prices. For the task of claim detection, we use a dataset developed by Shah et al. (2022a). The dataset contains binary labels (“in-claim”, and “out-of-claim”) for sentences extracted from a heterogenous set of analysts’ reports. In this case, the term “in-claim” text is used to refer to sentences in the financial domain that contain specific and measurable financial claims, which are not factual statements. For instance, the sentence “Operating income is expected to be between $2.1 billion and $3.6 billion” is considered an ”in-claim” sentence because it predicts a future outcome. On the other hand, the sentence “Revenues increased by 48.6% compared to the previous year, reaching $5.44 billion, primarily due to the expansion of the customer base” is categorized as “out-of-claim” since it presents factual information from the past.

Named Entity Recognition (NER) involves the identification and classification of named entities in text, such as person names, organization names, and locations, among others. Given a sentence with N tokens and a set of entities denoted as S with a size of #S, the NER model assigns an entity label to each token. Following the BIO convention, the label set L has a size of $2\#S+1$ and includes ”O” to indicate categories not listed in the entity list. NER plays a crucial role in financial NLP due to its significant implications for various applications in this domain. NER enables the extraction and categorization of key financial entities such as company names, person names, locations, etc. This facilitates the identification of important entities involved in financial news and reports, helping analysts and investors gain insights into market trends, company performance, and potential investment opportunities. For instance, NER can aid in tracking the mentions of specific companies in news articles, social media posts, and financial reports, enabling the monitoring of market sentiment and the detection of emerging patterns that may impact stock prices. Therefore, the accurate identification and classification of named entities through NER are crucial for extracting meaningful information and enhancing decision-making processes in financial NLP. For financial NER we use a dataset developed in Shah et al. (2023b).

In order to set the benchmark, we use base (“roberta-base”) and large (“roberta-large”) versions of RoBERTa (Liu et al., 2019) model. No pre-training is conducted on the models before proceeding with the fine-tuning process. To determine the optimal hyper-parameters for each model, a grid search is performed using four different learning rates (1e-4, 1e-5, 1e-6, 1e-7) and four different batch sizes (32, 16, 8, 4). We use a maximum of 100 epochs for training with early stopping criteria. If the validation F1 score doesn’t improve by more than or equal to 1e-2 in the next 7 epochs then we use the best model stored earlier as the final fine-tuned model. The experiments are carried out using PyTorch (Paszke et al., 2019) on an NVIDIA RTX A6000 GPU. The initialization of each model is based on the pre-trained version available in the Huggingface Transformers library (Wolf et al., 2020).

In the generative LLM category, we utilize ChatGPT (”gpt-3.5-turbo”) with specific settings including a maximum token limit of 1000 and a temperature value of 0.0. As for the open-source LLM category, we employ ”dolly-v2-12b” and ”h2ogpt-oasst1-512-12b” along with their dedicated text generation pipelines and models, which can be found on their respective Huggingface pages.

For each task, we design separate prompts and write functions that can be used to get labels from the output of the prompt. As zero-shot learning doesn’t require any labeled data for training, we only use test split for prompting. The prompt template for each task is discussed below.

Performance and other metrics for the FOMC tone classification task are reported in table 2. Fine-tuned RoBERTa-base and RoBERTa-large have similar performance and they outperform zero-shot LLMs. In the zero-shot LLM category, ChatGPT outperforms other open-source LLMs by a good margin. ChatGPT also follows the instruction 100% of the time, while Dolly fails to follow instructions for 6.05% of the time and H2O for 42.14% of the time. We run all the codes before we made the data for this task public as part of our other work (Shah et al., 2023a) to ensure that there is no contamination issue. The results for fine-tuning models are a little different than those reported in Shah et al. (2023a) as we employ different early stopping mechanism compared to them. Even if for a particular use case performance is not an issue, the time it takes for generative LLM to annotate the data is 1000 times higher compared to fine-tuned PLMs. The latency of a few seconds is not useful when markets might adjust the price in a few milliseconds.

For the sentiment analysis task, performance and other metrics are reported in table 3. The results follow a similar trend but for this task, the H2O model doesn’t follow the instruction at all which is surprising. It will be an interesting future study to understand why this is the case and how open-source LLMs can be improved in this dimension. The ChatGPT achieves impressive performance close to 0.9 F1 score for sentiment analysis which is not far from the performance of fine-tuned RoBERTa.

Performance and other metrics for the numerical claim detection task are reported in table 4. The gap between fine-tuned model PLMs and ChatGPT is larger here compared to the sentiment analysis task while the gap between Dolly and ChatGPT is smaller. The exact reason for this is uncertain, but one possibility could be contamination, considering that the sentiment analysis dataset is publicly available while the claim dataset is not. Exploring the potential contamination issues as an explanation for this discrepancy would be an intriguing avenue for future research. Other results follow a similar trend.

Table 5 presents the performance metrics for the Named Entity Recognition (NER) task. Similar to the previous tasks, the results exhibit a consistent pattern. However, the NER prompt proves to be more challenging compared to other tasks, leading both open-source models to struggle in adhering to the given instruction. Moreover, decoding labels from the prompt output is relatively more difficult in token classification compared to sequence classification tasks, resulting in a higher percentage of samples with missing labels for generative LLMs in the NER task.