RiskLabs: Predicting Financial Risk Using Large Language Model Based on Multi-Sources Data

The Audio Encoding component consists of a pre-trained audio embedding model, a Multi-Head Self-Attention, and an Average Pooling layer. Firstly, we extracted audio embedding using the Wav2vec2, a transformer-based Large Language Model proposed by Baevski et al. (2020). The Wav2vec2 model has been recognized for its effectiveness, showing that learning from speech audio alone can surpass some of the best semi-supervised methods in performance while maintaining conceptual simplicity. Wav2vec2 operates by masking the speech input in the latent space and engaging in a contrastive task based on a quantization of these latent representations, which are learned concurrently. Following this, the audio embeddings are processed through the Multi-Head Self-Attention mechanism. This step is crucial for distilling specific audio features, which are then poised for integration with features from other data modalities, ensuring a comprehensive and nuanced analysis.

To describe the Audio Encoding in more detail, we let the raw audio input data be represented by $A_{c}=\{a_{c}^{1},a_{c}^{2},\ldots,a_{c}^{n}\}$ where $a_{c}^{i}$ represents the $i^{th}$ audio frame in one data sample. Each audio frame will be converted into a vector representation:

Therefore, we obtain the audio embeddings $E_{ac}=\{e_{ac}^{1},e_{ac}^{2},\ldots,e_{ac}^{n}\}$. The shape of the $E_{ac}$ is 520 $\times$ 512, where 520 is the maximum number of audio files amongst all companies and 512 is the dimension of the resulting transform from the model for a single audio frame. Earnings conference calls with less than 520 audio frames ($n<520$) have been zero-padded for uniformity in input matrix size.

Then, we fed audio embedding $E_{ac}$ into the Multi-Head Self-Attention (MHSA) to extract the audio feature. The MHSA further comprises a multi-head attention block, followed by a norm block, and an MLP block. MLP denotes a two-layer feed-forward network (FNN) with a ReLU activation function. The MHSA block essentially forms the basis of all the architectures discussed later in the paper. In detail, the multi-head self-attention calculation process is as follows:

where $Q$ (queries) and $K$ (keys) of dimension $d_{k}$ and $V$ values of dimension $d_{v}$. The dimensions of the weights are: $W_{i}^{Q}\in R^{d_{model}\times d_{k}}$, $W_{i}^{K}\in R^{d_{model}\times d_{k}}$, $W_{i}^{V}\in R^{d_{model}\times d_{v}}$, and $W^{o}\in R^{d_{v}\times d_{model}}$. The dot product is then calculated for the query with all the keys. These values are then normalized by dividing each value by $\sqrt{d_{k}}$ and then we apply a softmax function to obtain the weights of the values:

The attention function on a set of queries is calculated simultaneously packed together in a matrix Q. The keys and values are also packed in the matrices K and V respectively.

Combining (2)-(4), we apply this process to $E_{ac}$:

where $T_{ac}=\{t_{ac}^{1},t_{ac}^{2},\ldots,t_{ac}^{n}\}$ with size 520 $\times$ 512. Following this, an average pooling layer is applied to $T_{ac}$:

where $T_{a}$ denotes the resultant extracted audio feature of size 512.

The overall process is similar to Audio Encoding. We first use SimCSE Gao et al. (2021) to extract the vector representation of each sentence in earnings conference transcripts. SimCSE is a sentence-level representation as word/token-level representations are too difficult and document-level embedding is too sparse and loses a lot of information. SimCSE is a Siamese neural network architecture that learns to embed pairs of sentences into a shared space where similar sentences are mapped close together and dissimilar sentences are mapped far apart. We let raw transcripts as $T_{c}\ =\{t_{c}^{1},a_{c}^{2},\ldots,t_{c}^{n}\}$ where $t_{c}^{i}$ represents the $i^{th}$ sentence in the transcript. Therefore, each sentence will be converted into a vector representation:

We obtain the corresponding text embeddings given by $E_{tc}=\{e_{tc}^{1},e_{tc}^{2},\ldots,e_{tc}^{n}\}$. The shape of the $E_{tc}$ is 520 $\times$ 768, where 520 is the maximum number of sentences amongst all data samples and 768 is the dimension of the output of SimCSE. Earnings conference calls with less than 520 sentences ($n<520$) have been zero-padded for uniformity in input matrix size.

Same with (2)-(5), the MHA is applied to $E_{tc}$ to get $T_{tc}=\{t_{tc}^{1},t_{tc}^{2},\ldots,t_{tc}^{n}\}$ with dimension 520 $\times$ 768. Then, the average pooling layer is applied to $T_{tc}$:

where $T_{t}$ denotes the resultant extracted textual feature of size 768.

An Earnings Conference Call is a teleconference or webcast in which a public company discusses its financial results for a reporting period, typically a quarter or fiscal year. During the call, senior company executives, including the CEO and CFO, present key financial figures, performance metrics, and strategies. They also provide insights into the company’s future outlook, including forecasts and potential challenges. Following the presentation, analysts and investors have the opportunity to ask questions.

In additional of the document structure, Earnings Conference Call within the same sector often exhibits similar discussion topics. For instance, in the energy sector, topics frequently revolve around oil and gas price fluctuations, regulatory changes, and environmental sustainability efforts. Earnings Conference Call in the consumer sector typically centers around consumer behavior trends, e-commerce growth, supply chain management, and product innovation.

To obtain insights from an Earnings Conference Call on how it might influence future market volatility, our approach encompasses two primary steps. (See Figure 2) Initially, we summarize the entire earnings conference call to capture a broad understanding of the company’s performance and future outlook. Following this, the second step involves a meticulous examination of specific points of interest to investors, for example focusing on detailed financial metrics and performance indicators mentioned in the call.

However, when attempting to summarize or retrieve information from earnings conference call transcripts using LLMs, directly inputting the entire document can pose significant challenges. The lengthy nature of these documents, often extending to thousands of words, along with the inclusion of sentences ranging from crucial insights to irrelevant remarks such as greetings, can affect the LLM’s ability to effectively comprehend and analyze the content of the earnings conference call.

Therefore, in the proposed Earnings Conference Call analyzer, it begins by segmenting the call transcripts based on the average length attributed to each topic. By organizing the text in this manner, we create distinct embeddings for paragraphs that revolve around similar ideas or themes. This strategy significantly enhances the model’s efficiency in information processing at later stages.

After dividing the lengthy earnings conference call transcript into several smaller sections, we use LLM to summarize each individual chunk. Since the content within each chunk is more homogeneous and concentrated around a single theme, the LLM can more effectively grasp the core idea. (See Figure 2 Green part) At the same time, to further combine the analyzed content with other extracted features, we use LLM to extract embedding from the analyzed earnings conference call as the analysis feature:

After generating summaries for each segmented chunk of the earnings conference call, we proceed to aggregate these individual summaries and make a second summarization again. The initial summaries distill the essence of each chunk, stripping away extraneous details and focusing on the core information. When these condensed summaries are combined and summarized again, the reduced length facilitates an even more efficient and effective summarization by the LLM. Consequently, this two-tiered approach to summarization not only enhances the manageability of the text for the LLM but also ensures that the final summary is both comprehensive and succinct, capturing the most critical insights in a more accessible format.

In addition to summarization, the earnings conference call analyzer delves deeper into analyzing the transcript. We simulate the analytical process of financial analysts as they examine an earnings conference call. Experts will prioritize certain information, meticulously analyzing key financial metrics and performance indicators to gauge the company’s current state and future prospects.

To simulate this analysis process, we begin by identifying core topics analysts typically focus on in earnings conference call, such as dividends, earnings, and costs. Then we use LLM to locate and extract paragraphs mentioning these topics. However, accurately extracting this information from the dense narrative of earnings conference calls can pose a challenge. Because LLM searches are based on similarity, comparing a single-word topic to an entire document presents a scale issue, and a single word may lack the necessary context, thus diminishing the extraction capability. To enhance the precision of information extraction, we formulate a variety of questions related to each topic but phrased differently, ensuring a broad coverage and increasing the likelihood of accurately pinpointing relevant sections. For instance, regarding the dividends topic, we pose questions such as: (1) "Have there been any changes in the stock dividends, and if so, at what rate have the dividends increased or decreased?" (2) Does the company Board expect to increase the stock dividends in the next future? (3) How does the dividend yield compare to the peers?

After locating the segments where these topics are discussed, we extract the corresponding paragraphs. These paragraphs, however, may still contain irrelevant text such as preceding discussions. To address this, we use a contextual compression method from LLM, adept at distilling the paragraph down to its most relevant sentences, effectively obtaining the crucial data from the noise.

Our subsequent step involves determining how the company’s present position of each topic could influence future stock behavior. To this end, we consult financial experts to provide various potential scenarios within a specific topic and offer their professional insights on how each scenario could impact market responses. For example, regarding the dividend topics, the earnings conference call may discuss about increasing dividends and decreasing dividends in these two scenarios. Increasing dividends may drive positive returns and lower stock volatility. Conversely, a decrease in dividends might be perceived negatively by investors, leading to decreased returns and increased volatility in the stock’s price. Once again, we face the challenge of mapping sentences from earnings conference calls to various potential scenarios. Our approach involves a detailed, sentence-by-sentence comparison. (See Figure 2 Yellow part) We collect various expressions from multiple earnings conference calls, focusing on how speakers convey specific scenarios, like the increase of dividends. In this way, we can precisely determine the current scenario of the company, and then, guided by expert insights, we can gain an understanding of how the stock is likely to move in the coming days.

In summary, our analysis of the earnings conference call includes two parts: summarizing the documents to obtain an overview and extracting precise information for in-depth company analysis. Ultimately, we merge these two components to form an integrated earnings conference call encoder.

Given the model’s reliance on several inputs and diverse data types, we identify an effective fusion structure to integrate these features into the training process to ensure a balanced weighting among components. We use additive interactions to handle the representational fusion of different abstract representations. These operators can be viewed as differentiable building blocks that combine information from several different data streams and can be flexibly inserted into almost any unimodal pipeline Liang et al. (2022). Given the audio feature $T_{a}$, textual feature $T_{t}$ from the transcript, and $T_{l}$ from ECC analyzed text, additive fusion can be seen as learning a new joint representation:

where $w_{1}\in R^{512\times 512}$, $w_{2}\in R^{768\times 512}$ and $w_{3}\in R^{1024\times 512}$ are the weights learned for additive fusion, $w_{0}$ the bias term and $\epsilon$ the error term. $E$ is a vector with 512 as the final feature from the Earning Conference Call Encoder.

For our time series input, we capture the VIX values from a 30-day period preceding the ECC release date. To distill meaningful features from this VIX series, we employ a Bidirectional Long Short-Term Memory (BiLSTM) network Siami-Namini et al. (2019):

In this setup, the BiLSTM is configured with 64 hidden states. Consequently, the output $T_{v}$, representing the extracted time series features, has a dimensionality of 128, reflecting the bidirectional nature of the BiLSTM.

The task of RiskLabs is to predict multiple risk metrics in the meantime. Relationships will exist among them. Therefore, by identifying and modeling the relationships between them, we can significantly improve the accuracy of our predictions. This method shifts the focus from treating each metric as an isolated entity to understanding them as part of a complex system, where the dynamics between variables can provide critical insights for more precise risk assessment.

To achieve this, we attempt to find the relationship that links the two different stages: the $m$-day before and the $m$-day after. We will use the VAR(Vector Auto-Regression Model) to capture the linear relationship between the volatility of different terms. We set four different "$m$-day volatility"(3-day volatility, 7-day volatility, 15-day volatility, and 30-day volatility). We define the $\sigma_{-m,t}$ as the "$m$-day future volatility" at time $t-m$; the calculation method traces back $m-day$ from today and calculates the $m-day$’s standard deviation of returns.

Volatility typically shows the cluster characteristics, which means that the change transmission pattern to the next stage tends to be similar to the last transmission pattern. These characteristics indicate that we could use the results from the close historical data as indicators or predictors to predict the next stage. We assume that the correlation between the volatility among different time scales will affect each other. We also measure the correlation by estimating the coefficient matrix and the information included in the historical data.

To estimate the coefficient of the model, we use the Bayesian Methodology by estimating the posterior distribution of the coefficient. If we have a general linear regression $Y=X\theta$, here $\theta$ is the coefficient that we need to estimate. We can have the:

Here, $P(\theta)$(or $P(\theta|X)$) is the prior distribution that we could guess from the existing research or common knowledge. We also suppose that the independent variable is independent with the regression coefficient; $P(Y|\theta,X)$ is the Likelihood Estimation from the data, and $P(Y|X)$ is the marginal distribution. Usually, we could view $P(Y|X)=\int_{-\infty}^{\infty}P(Y|X,\theta)p(\theta)d\theta$ as a constant because we could have the real value from the true dataset. From the Equation 12, we could have the following relation:

If we can determine the prior and the likelihood function, we can estimate the posterior distribution. We assume that the prior distribution $P(\theta)$ satisfies the Normal distribution based on the simple linear regression. Subsequently, we consider the Likelihood $P(Y|\theta,X)$. In the empirical study from Andersen et al. (2001), the unconditional distribution of the realized historical distribution of the volatility is highly right-skewed. Besides, Cizeau et al. (1997) tested the distribution historical volatility of S&P 500 stock from 1984 to 1996 and demonstrated that the pattern follows the log-normal distribution. Therefore, we took the log form of the original data. Given the preparation, we build the VAR model as follows:

Here, $\sigma_{m,t}$ stands for the "$m$-day" future volatility at time $t$; $u_{m,t}$ is a White noise term, which $u_{m,t}\sim N(0,1)$; $\beta_{i,j}$ represents the linear relationship, which is the coefficient matrix of the VAR model, $\alpha_{m}$, which is the intercept term of the VAR model.

We apply the Monte Carlo Markov Chain(MCMC) algorithm to obtain the posterior distribution of the coefficient. MCMC is a method that is based on Bayesian Estimation. It is mainly used to predict the posterior distribution in a probability space by sampling. The Markov Chain follows the equation:

This could be explained by the fact that the probability of transforming only depends on the former statement. And based on the convergence theorem of Markov Chain, suppose the Markov Chain converges to one probability at $n$th step: $\pi_{n}(x)\rightarrow\pi(x)$, and $X_{n}\sim\pi(x)$, $X_{n+1}\sim\pi(x)$, $...$, we could say that $X_{n}$, $X_{n+1}$, $...$ are i.i.d random variables.

We attempt to calculate the expectation of the posterior distribution by sampling. However, the posterior distribution is challenging to measure. To achieve this, we introduce the MCMC method to create a Markov Chain to achieve a stationary distribution and make it our desired posterior distribution. For MCMC, Metropolis-Hastings introduced a system to find a Markov Chain. For the model with many parameters to estimate, another alternative way is to use the Gibbs Sampling to update the estimated parameters one by one.

Regarding testing the MCMC process, we use several matrices, such as Monte Carlo standard errors (MCSE). There are two types, which measure the standard error of the mean and the standard error of the standard deviation of the chains. Also, because we did not generate the independent samples, the simulated samples are correlated. We want to know how many theoretically independent samples we drew and create the effective sample size (ESS), which measures the number of effectively independent samples we draw. Besides, we need to test if we achieve the stationary of our chains. Gelman-Rubin R-hat statistic($\hat{R}$) gave us a measure. We measure $\hat{R}$ by:

Usually, $\hat{R}$ should close to 1 and less than 1.01 otherwise it doesn’t achieve the stationary with the Markov Chain.

In order to validate this methodology, we choose one of the stocks "TWTR," in our database as a case study, and then output the Bayesian regression results. Here, we used 250 days of data(from 2016-02-22 to 2017-02-15) as the model training period. Table 1 shows the training result. We observe that the $\hat{R}$ of all parameters is equal to 1, and the $ess\_tail$ value and $ess\_bulk$ are all over 7000. Typically, there is no specific argument about the number of ESS. Kruschke (2014) in the book mentioned that we need more ESS samples for the interested parameters. In the meantime, we also need to balance the number of ESS data and model training costs. Vehtari et al. (2021) in the paper recommends having the ESS number over 400, which should be dependent on practical experience and model, to get a stable Monte Carlo standard error.

We used the rolling window(see Figure 6) to make a rough simulation only based on the historical data of different volatility terms. We chose 250 days(Approximately one year) as the training set window for the next-day prediction and iterated 100 times. The result is shown in Table 2. This table measures the distribution of AEP(Absolute Error Percentage), which is calculated by the following formula:

Here, the $\hat{y}$ is the estimated value from the Bayesian VAR model, and $y$ is the true value. From Table 2, we can see that the average estimation bias will become smaller when we estimate the longer volatility term, the same as the standard deviation of AEP. The negative skewness tells us that the peak of the AEP distribution is close to the y-axis(Because the mean is negative, as we calculated). In general, this experiment shows that using the VAR model based on historical data to predict the future term should be possible and could maintain some accuracy.

We compare our approach to volatility prediction to several important baselines as described below.

• •

  Classical Methods: The analysis incorporates the GARCH model, a classical auto-regressive model for predicting volatility, as well as its various derivatives Franses and Van Dijk (1996); Kim and Won (2018). These models are widely recognized and commonly employed in the realm of volatility prediction. Primarily, they are tailored for short-term volatility forecasting and may not perform as efficiently in predicting average volatility over an extended period (e.g., n-day volatility).

• •

  LSTM Gers et al. (2000): Long Short-Term Memory Networks (LSTMs) are a popular choice for financial time series prediction due to their efficacy in handling sequential data. In the context of volatility prediction, we select a straightforward LSTM model as a benchmark.

• •

  MT-LSTM+ATT Luong et al. (2015): merges the prediction of average n-day volatility with the forecasting of single-day volatility, using attention-enhanced LSTM as the foundational learning models.

• •

  HAN (Glove): This baseline implements a Hierarchical Attention Network that incorporates dual-layered attention mechanisms at both the word and sentence levels. Initially, every word within a sentence is transformed into a word embedding through the pre-trained Glove 300-dimensional embeddings. Subsequently, these embedded sentences are processed by a Bi-GRU encoder Chung et al. (2014), and in parallel, another Bi-GRU encoder is utilized to formulate a representation of each document as a series of sentences. This representation of the document is then fed into the concluding regression layer to generate predictions.

• •

  MRDM Qin and Yang (2019): The MRDM model first introduced a multi-modal deep regression approach for volatility prediction tasks. It utilizes pre-trained GloVe embeddings and bespoke acoustic features, which are processed through individual BiLSTMs to generate uni-modal contextual embeddings. These embeddings are subsequently merged and input into a two-layer dense network for further processing.

• •

  HTML Yang et al. (2020): This work presented a state-of-the-art model that employs WWM-BERT for text token encoding. Similar to MDRM, HTML also leverages the same audio features. These unimodal features are then combined and processed through a sentence-level transformer, resulting in multimodal representations for each call.

• •

  GPT-3.5-Turbo: We evaluated the efficacy of using GPT-3.5-Turbo for direct financial risk prediction. The input for this test was earnings conference calls, and we instructed GPT-3.5-Turbo to generate numerical risk forecasts based on the provided EC, utilizing a specified prompt setting. In the experiment, we set the temperature as zero.

We use GPT-4 for the Earnings Conference Call Analyzer and the News-Market Reactions Encoder, utilizing it to analyze ECC data and build the news database. Throughout this process, we set the temperature parameter to 0. This ensures that the Large Language Models (LLMs) produce the most predictable responses, which aids in maintaining consistency in our experiments.

For dataset usage, we strategically partitioned it into a training set and a test set, adhering to an 8:2 ratio. It is crucial to highlight that the division of the data was conducted on a temporal basis, ensuring that the dates of the data in the training set always precede those in the test set. This temporal segregation is imperative for maintaining the integrity of our predictive model. By structuring the dataset in this manner, we ensure that the training process is consistently oriented towards predicting future risks based on past data, a fundamental principle for the accuracy and reliability of our forecasting methodology.

For the overall training of the framework, we developed the code using PyTorch. Each Multi-Head Attention layer in the network comprises 8 individual heads, and the training process utilized batch sizes $b\in\{2,4,8\}$. We use a grid search to determine the optimal parameters and select the learning rate $\lambda$ for Adam optimizer among $\{1e-3,1e-5,1e-6,1e-7\}$. The best hyper-parameters were kept consistent across all experiments, with the exception of the trade-off parameter $\mu$ which varied between the two tasks.