Enhancing Financial Sentiment Analysis with Expert-Designed Hint

Financial sentiment analysis, which classifies given text into bullish or bearish categories, has long been a topic of research in NLP Baker and Wurgler (2007); Liu (2015); Xu and Cohen (2018). Here, bullish (bearish) indicates the investor’s expectation that the price of the mentioned stock will rise (decline). When dealing with financial textual data, numbers have often been highlighted. Some studies attempt to extract numerical information Chen et al. (2019a, b). Others focus on the role of numbers in financial documents to explore reasoning skills Zhu et al. (2021); Chen et al. (2021); Nan et al. (2022). Further research suggests that the attention to numbers in financial documents can enhance downstream task performance, such as volatility forecasting Yang et al. (2022); Shi et al. (2023). However, the relationship between financial sentiment analysis and numbers in text is rarely discussed. Building on this premise, this paper examines the efficacy of prompting LLMs to consider expert-designed hints, specifically the importance of numbers in understanding financial documents. Our findings indicate that the implementation of “expert-designed hints,” such as emphasizing the relevance of numerical data, substantially enhances the performance of sentiment analysis across various LLMs.

Financial sentiment analysis has been a longstanding topic in Natural Language Processing (NLP) research. Some studies focus on the writer’s self-annotated labels, which are provided by the writer when posting tweets Li and Shah (2017); Xing et al. (2020). Others make predictions about the readers’ sentiment Agić et al. (2010); Yuan et al. (2020); Gaillat et al. (2018). Additionally, there are studies that discuss the difference between writers’ and readers’ sentiment Maks and Vossen (2013); Berengueres and Castro (2017). For example, Chen et al. (2020) show that writers’ and readers’ financial sentiments may differ based on a survey of 10K financial tweets. This discrepancy provides a basis for discussing perspective-taking within sentiment analysis, specifically, the extent to which models, acting as readers, can deduce a writer’s sentiment, especially when it is implicitly conveyed. This paper aims to bridge this gap by utilizing the Fin-SoMe dataset Chen et al. (2020), comprising approximately 10,000 annotated social media posts from both the writer’s and reader’s viewpoints. Our results suggest that the expert-designed hint activates perspective-taking and further enhances the performance of financial sentiment analysis.

In sum, this short paper aims to answer the following three research questions (RQ):

Theory of Mind (ToM) Baron-Cohen (1997); Baron-Cohen and Hammer (1997); Barnes-Holmes et al. (2004); Baron-Cohen et al. (2013) has been a subject of interest for a long time and has recently regained attention. Discussions from the perspectives of vision Liu et al. (2022), gaming Li et al. (2023), and psychology van Dijk et al. (2023) are provided, but discussions in sentiment analysis are scarce. Perspective-taking involves understanding others’ thoughts by empathizing with their viewpoints, a concept extensively explored within the realm of psychology under the ToM. LLMs have shown remarkable capabilities in grasping the semantics explicitly expressed in texts. Consequently, researchers in the NLP community have begun to investigate the applicability of ToM in evaluating whether models can empathize with human perspectives Liu et al. (2022); Li et al. (2023); van Dijk et al. (2023); Sclar et al. (2023); Sileo and Lernould (2023). Despite numerous studies replicating psychological experiments to assess language models’ abilities, investigations into perspective-taking for enhancing understanding of sentiments on social media are scant. We propose a novel task design based on an existing dataset and discuss perspective-taking in financial sentiment analysis.

We use the Fin-SoMe dataset Chen et al. (2020) in our experiments. A total of 10,000 tweets were collected from the social media platform Stocktwits,¹¹1https://stocktwits.com/ each labeled by the post’s writer as either bullish or bearish. Reader sentiment was gauged by asking annotators to classify each tweet based on its content into bullish, bearish, or none categories. Tweets that did not explicitly convey sentiment were labeled as none. To discuss perspective-taking ability, instances that received a bullish or bearish label from the post’s writer but were not labeled as bullish or bearish by the reader were deemed to require perspective-taking ability in this study. The dataset statistics is presented in Table 1. This research does not involve training models; the entire dataset is treated as a test set to assess models’ proficiency in financial sentiment analysis. The experimental results from this dataset inform the discussions of RQ1 and RQ2.

Our experiments aim to ascertain if LLMs inherently possess the ability to recognize and utilize the nuanced hint that experts deploy in deciphering financial documents. Furthermore, we discuss the ability of LLMs to require perspective-taking ability. We posit that if LLMs are inherently knowledgeable of and can apply these nuanced hints, their performance in sentiment analysis tasks should remain consistent, irrespective of whether these hints are explicitly highlighted within the prompt. Conversely, a notable discrepancy in performance would imply LLMs’ lack of intrinsic capability to leverage these nuanced hints without explicit expert instruction.

Furthermore, our methodology incorporates not only the direct sentiment prediction approach, termed as the Simple Prompt, but also includes the Chain-of-Thought (CoT) technique Wei et al. (2022), which involves eliciting an analytical reasoning before concluding with a prediction. To evaluate the potential enhancement in performance through the integration of nuanced hints, we augment our prompts with a directive emphasizing the importance of numerical data or statistics embedded within the tweets, as these elements could be pivotal in deducing the overarching sentiment. This approach is inspired by numerous studies on number comprehension and reasoning within financial texts Zhu et al. (2021); Chen et al. (2021); Nan et al. (2022); Yang et al. (2022); Shi et al. (2023), which collectively suggest that numerical information plays a critical role in understanding intentions or making informed decisions. In short, we add a sentence, “Focus particularly on any numerical data or statistics present in the tweet, as these figures may be crucial in determining the overall sentiment.”, in the CoT prompt as a hint in the experiment.

Our analysis encompasses four LLMs, including PaLM 2 Anil et al. (2023), Gemini Pro,²²2https://deepmind.google/technologies/gemini/ GPT-3.5, and GPT-4.³³3https://platform.openai.com/docs/models To quantify their performance, we employ both Micro-F1 and Weighted-F1 scores. Furthermore, we utilize McNemar’s test McNemar (1947) to ascertain the statistical significance of performance disparities among the models, setting the significance threshold at $\alpha=0.05$.

This study investigates the impact of employing expert-designed hint on the performance of LLMs in financial sentiment analysis. We find that LLMs do not inherently utilize subtle hints crucial for sentiment analysis without explicit instruction. Introducing a simple, expert-derived hint that highlights the significance of numerical data substantially improves the models’ capability to identify sentiments. This enhancement is especially notable in scenarios requiring perspective-taking, where the models must deduce the sentiment implied by the writer, highlighting the necessity of explicit guidance in financial sentiment analysis tasks.

The limitations of this study are discussed as follows.

Firstly, the findings of this study are primarily based on financial social media data, particularly from the Stocktwits platform. This focus may limit the generalizability of our conclusions to other domains or types of social media content. Future studies could explore whether the observed benefits of perspective-taking and expert-designed hint extend to other domains, such as healthcare or politics, where sentiment analysis is equally critical.

Secondly, this study simplifies the concept of perspective-taking. However, perspective-taking in human communication is a complex, multi-dimensional process that involves understanding emotional states, intentions, and contextual factors. Future work could aim to model these additional layers of complexity to achieve a more holistic understanding of sentiment in social media texts.

Another limitation is the focus on only four LLMs in our experiments. While these models are among the most advanced at the time of our study, the rapidly evolving field of natural language processing continually introduces new models that may offer different insights into the challenges of financial sentiment analysis. Testing our approach with a wider array of LLMs could provide a more comprehensive understanding of its effectiveness.

Lastly, our study’s focus on numerical data as a key element of financial sentiment analysis may overlook other important factors that influence sentiment interpretation, such as linguistic subtleties, cultural references, or domain-specific knowledge. Incorporating these dimensions into future research could provide a more holistic understanding of the challenges and opportunities in applying large language models to financial sentiment analysis.