Forecasting S&P 500 Using LSTM Models

The Autoregressive Integrated Moving Average (ARIMA) model is highly used as a statistical approach for forecasting sequential data. As demonstrated by Adebiyi et al. [4], the ARIMA model can capture short-term dependencies and generate a reasonable prediction, which was used to forecast financial data.

The ARIMA model consists of three parts:

• •

  Autoregressive (AR): The linear relationship between its current value and its lagged observations.

• •

  Differencing (I): The number of times the series is differenced to achieve stationarity to remove trends and seasonality.

• •

  Moving Average (MA): Relates the current value to past forecast errors using a number of lagged errors.

The study by Adebiyi et al. applied ARIMA to predict daily stock prices and found that the model performed well on stationary time series. Their experiments involved rigorous parameter selection (using $p$, $d$, and $q$ terms) and model validation. They also acknowledged that ARIMA struggles with capturing the nonlinear and long-term dependencies with the given data set, which are rooted in the stock price’s movements.

While ARIMA focuses on the average behavior seen throughout the time series data, factoring in volatility is essential for forecasting financial data as it factors into the prices of the stock market. Autoregressive Conditional Heteroskedasticity (ARCH) and its generalized form Generalized ARCH (GARCH) models address this by explicitly modeling for time-varying volatility.

Raheem et al. [11] explored ARCH/GARCH models to model volatility in financial time series. These models assume that variance at a given time depends on the squared residual errors of previous periods (ARCH) or combines past residuals with previous conditional variances (GARCH). Their model showed effectiveness in capturing volatility clusters, where it was seen that large changes in the stock price were followed by even larger changes, and small changes tend to follow small changes.

The mathematical form of a GARCH(1,1) model can be represented as:

where $\alpha$ is the ARCH parameter indicating how sensitive today’s conditional volatility, $\sigma_{t}^{2}$, to the prior day’s squared residual $\epsilon_{t-1}^{2}$. $\beta$ is the GARCH parameter, showing the persistence of the past volatility to its current $\sigma_{t}^{2}$.

Raheem et al. [11] applied GARCH to stock market data and showed that it has strengths in forecasting volatility, which is crucial for pricing options and risk management. They also noted that the models are sensitive to outliers and they assume a symmetric response to shocks (positive or negative impacts), which is not always the case for the financial market.

Support Vector Regression (SVR) is a machine learning method that is popular for its use in handling high dimensional and nonlinear data. Wang [2] showed how this model combined with Grey Correlation Degree (GCD) can be used to enhance accuracy by optimizing feature selection which improved stock price forecasting. GCD assigns weights to input features based on their correlation with its target variable. This allows SVR to focus on the most relevant features in order to make its prediction.

SVR is comprised of a hyperplane in a high-dimensional space which minimizes the error bet between the predicted and actual values while keeping the model as generalized as possible. By introducing GCD, enhancements are made to SVR by reducing noise and focusing solely on the critical predictors which addresses the key challenge in stock prediction: finding the most impactful features. By improving the feature selection, Wang [2] showed a significant boost in accuracy when applying it to the dataset. This preprocessing technique showed how well a model can perform given the right features it trains on.

The k-Nearest Neighbors (KNN) is an algorithm that predicts future values by finding past values that are most closely related to the current one. Alkhatib et al. [10] used the KNN algorithm to forecast stock prices for six major companies listed on the Jordanian Stock Exchange. The KNN algorithm measures the Euclidean distance between data points to identify the closest matches and uses these neighbors to make predictions. KNN does not make assumptions on the underlying data showing that it is a nonparametric method. Alkhatib et al. [10] showed that KNN was quite effective when predicting closing prices. This is because of its simple structure and its ability to adapt to nonlinear relationships within the data. During this study, the evaluation of KNN’s robustness for stock price prediction by measuring error ratios was seen to be low. The authors highlighted that selecting the right amount of neighbors and distance metrics is key for performing well. It is also worth noting that this model does struggle with very large datasets and noisy data if the right preprocessing is not performed.

Random Forest is a machine learning method that combines multiple decision trees to improve prediction accuracy and reduce overfitting. Khaidem et al. [12] applied Random Forest to predict the direction of stock market prices using historical stock data and technical indicators to forecast stock for major companies like Apple (AAPL) and General Electric (GE). Random Forest generates several decision trees, each trained on random subsets of features and data points. The predictions from individual trees are combined by averaging to produce a final output. Khaidem et al. [12] showed this method achieves higher prediction accuracy due to its ability to model nonlinear patterns in financial data.

Hidden Markov Models (HMM) are used widely for time series forecasting to model systems that transition between hidden states over time. Hasanbas [7] applied HMM to forecast financial time series to show its ability to capture the probabilistic states of stock market trends. In this application of the model, HMM assumes that the observed time series data are generated by a sequence of hidden states, each of which each follows a distinct probability distribution.

HMM consists of three parts:

• •

  Hidden States: Unobserved states that the system transitions between over time

• •

  Transition Probabilities: Probabilities of transitioning from one hidden state to another at each time step.

• •

  Emission Probabilities: Probabilities of observing a particular output given the hidden state.

For stock prediction, HMM identifies market regimes (e.g., bullish, bearish, or stable) by analyzing transitions between these states based on historical price movements. Hasanbas [7] showed that HMM effectively captures short-term dependencies, making it well-suited for a volatile stock market. It is important to note that HMM assumes Markovian properties, meaning its probabilities of transitioning to the next state only depend on its current state. This makes it less viable for forecasting long-term.

The Bayesian method is another probabilistic method used for time series forecasting. Unlike the previous model, the Bayesian model introduces uncertainty into the predictions. Steel [8] showed the application of Bayesian time series analysis for stock prediction. This model involves constructing posterior distributions for model parameters using Bayes’ theorem. For financial data, this model is well suited as it can handle noisy, incomplete, or volatile data. Steel [8] highlighted that Bayesian methods can outperform deterministic approaches in volatile markets by accounting for model and parameter uncertainties.

Long Short-Term Memory (LSTM) networks have been shown to be one of the best choices when it comes to handling sequential data. They are an improvement upon RNNs, which struggle with the vanishing gradient problem, where information from early time steps tends to fade away. Their unique architecture comprises memory cells along with three gates:

• •

  Input Gates: Controls which parts of the previous memory are discarded.

• •

  Forget Gates: Determines what new information should be added to the memory.

• •

  Output Gates: Regulates what information is output for the current step.

These components work together to selectively retain, update, and output information to capture both the short-term fluctuations and long-term trends in the dataset. Zou and Qu [1] implemented LSTM networks to predict stock prices and analyze this strategy against others. Their study showed that the LSTM model outperformed the traditional RNNs and machine learning methods by effectively learning patterns in historical stock price data. Sonkavde et al. [3] conducted a review comparing machine learning and deep learning methods which included LSTM, Convolutional Neural Networks (CNNs), and traditional RNNs. Their result showed the effectiveness of the LSTM model compared to RNNs and CNNs by handling temporal and spatial features better.

Another improved variant of RNNs is called $\alpha$-RNN, which offers some computational efficiency while being able to maintain good prediction accuracy. Dixon and London [9] proposed using $\alpha$-RNN for forecasting financial time series data. The $\alpha$-RNN enhances traditional RNNs by introducing an exponential smoothing layer which helps the model retain long-term information in the data by combining past and current states. Dixon and London showed that $\alpha$-RNNs are competitive in financial forecasting as they offer a reduced computational overhead when compared to LSTM. However, they lack fine-grained control over information flow which allows LSTM to handle complex, long-term dependencies.

Exploring spatial and relational information in financial data has led some researchers to use Graph Neural Networks (GNNs). GNNs are designed to process data that is represented as graphs, where each node and edge encode entities and their relationships. Xiang et al. [6] proposed using a variant of GNNs called Temporal and Heterogenous GNN (TH-GNN). This allowed them to try and forecast financial time series by modeling a relationship between several financial indicators and the market data. Unlike the sequential model LSTM, this model captures dependencies between different entities, such as different stocks and indices, over time providing a better understanding of the market.

Another emerging trend in deep learning is integrating Federated Learning (FL) and Large Language Models (LLMs) for financial forecasting. FL is a decentralized machine learning technique that allows multiple edge devices (clients) to collaboratively train a global model while keeping the data localized. LLMs are advanced machine-learning models designed to understand and generate human-like text. They are built using the transformer architecture, where text is converted into numerical representations called tokens, which are then processed through a series of encoder and/or decoder layers that analyze relationships between words in a sentence. Abdel-Sater and Hamza [5] introduced an FL model which incorporated a pre-trained LLM. Instead of processing text, Abdel-Sater and Hamza adapted the LLM to handle stock data. This allowed them to use the collaborative training on multiple data sources without sharing any raw financial data. However, this comes with the cost of being less efficient on numerical financial data as opposed to the LSTM model.