An Evaluation of Standard Statistical Models and LLMs on Time Series Forecasting ^††thanks: 979-8-3503-7565-7/24/$31.00 ©2024 IEEE

A language model is created to determine the probability that a specific sequence of words forms a coherent sentence. For example, let’s consider two instances: Sequence A = never, too, late, to, learn, which clearly constructs a sentence (”never too late to learn.”), and a proficient language model should assign it a high likelihood. Conversely, the word sequence B = ever, zoo, later, too, eat is less probable to create a coherent sentence, and a well-trained language model would assign it a lower probability. The main objectives of language models involve evaluating the probability of a word sequence forming a grammatically correct sentence and forecasting the probability of the next word based on the preceding words. To accomplish these objectives, language models work under the fundamental assumption that the occurrence of a subsequent word is influenced only by the words that come before it. This assumption allows the sentence’s joint probability to be related to the conditional probability of each word in the sequence. Language models are trained on a set of sequences denoted as $U={U_{1},U_{2},\ldots U_{i},\ldots,U_{N}}$, where each sequence is depicted as $U_{i}=(u_{1},u_{2},\ldots,u_{j},\ldots,u_{n_{i}})$, and each token $u_{j}$ ($j$ in $\{1,2,\ldots,n_{i}\}$) belongs to a vocabulary $\mathcal{V}$. Large language models typically function as autoregressive distributions, implying that the probability of each token is dependent solely on the preceding tokens in the sequence, and can be formulated as $\begin{aligned} p_{\theta}\left(U_{i}\right)=\prod_{j=1}^{n_{i}}p_{\theta}\left(u_{j}\mid u_{0:j-1}\right)\end{aligned}$. In this context, the model parameters $\theta$ are acquired by maximizing the probability of the complete dataset, expressed as $\begin{aligned} p_{\theta}(\mathcal{U})=\prod_{i=1}^{N}p_{\theta}(U_{i})\end{aligned}$.

LLMs represent a type of artificial intelligence model created to comprehend and produce human language. They undergo training on extensive text datasets and have the capability to carry out various tasks, such as text creation [17, 18, 19], machine translation [20], dialogue systems [21], among others. These models are distinguished by their immense size, encompassing billions of parameters that facilitate their understanding of intricate patterns in language-related data. Typically, these models are constructed on deep learning frameworks, like the transformer architecture [22], which contributes to their remarkable performance across a spectrum of NLP assignments. The development of Bidirectional Encoder Representation Transformers (BERT) [11] and Generative Pre-trained Transformers (GPT) [12] has significantly propelled the NLP domain, ushering in the widespread adoption of pre-training and fine-tuning techniques [23][24]. Extensive datasets are leveraged through task-specific aims and are commonly denoted as unsupervised training (although technically supervised, it lacks human-annotated labels) [25][26]. Subsequently, it is standard procedure to fine-tune the pre-trained model to amplify the utility of the final model for downstream tasks. Noteworthy is the superior performance of these models over earlier deep learning approaches [27], prompting a focus on the meticulous design of pre-training and fine-tuning procedures. Research endeavors have also shifted towards the ultimate objectives of machine learning, pre-training language models as intermediary components of self-directed learning tasks.

The various functions associated with time series can be categorized into multiple types, including prediction, anomaly identification, grouping, identification of change points, and segmentation. Prediction and anomaly detection are among the most commonly utilized applications. The task of time prediction involves examining the historical data patterns and trends in time series information to forecast future numerical changes using different time series prediction models. The forecasted future time point can be a singular value or a series of values over a specific time frame. This predictive task is frequently employed to analyze and predict time-dependent data such as stock prices, sales figures, temperature variations, and traffic volume. Given a sequence of values $Y=\{x_{1},x_{2},\ldots x_{i},\ldots,x_{N}\}$, where the value corresponding to time stamp $t$ is $x_{t}$, the objective of prediction is to estimate $x_{t+1}$ based on $\{x_{1},x_{2},\ldots,x_{t}\}$.

Traditional time series modeling, such as the ARIMA model, is commonly employed for forecasting time series data. The ARIMA model consists of three key components: the autoregressive model (AR), the differencing process, and the moving average model (MA). Essentially, the ARIMA model leverages historical data to make future predictions. The value of a variable at a specific time is influenced by its past values and previous random occurrences. This implies that the ARIMA model assumes the variable fluctuates around a general time trend, where the trend is shaped by historical values and the fluctuations are influenced by random events within a timeframe. Moreover, the general trend itself may not remain constant. In essence, the ARIMA model aims to uncover hidden patterns within the time series data using autocorrelation and differencing techniques. These patterns are then utilized for future predictions. ARIMA models are adept at capturing both trends and temporary, sudden, or noisy data, making them effective for a wide range of time series forecasting tasks.

If we temporarily set aside the distinction, the ARIMA model can be perceived as a straightforward fusion of the AR model and the MA model. In a formal manner, the equation for the ARIMA model can be represented as:

In the equation provided, $x_{t}$ represents the time series data under examination, while $\varphi_{1}$ to $\varphi_{p}$ denote the coefficients of the autoregressive (AR) model that depict the connection between the present value and the value at the previous $p$ time instances. Similarly, $\theta_{1}$ to $\theta_{q}$ stand for the coefficients of the moving average (MA) model, which elucidate the relationship between the current value and the deviation at the last $q$ time points. The term $\epsilon_{t}$ symbolizes the deviation at time $t$, and $c$ is a constant term.

A technique for predicting time series using Large Language Models (like GPT-3) involves converting numerical data into text and generating potential future trends through text predictions. Essentially, this approach converts time series data into a sequence of numbers and interprets time series prediction as forecasting the next token in a textual context, leveraging advanced pre-trained models and probabilistic functions like likelihood evaluation and sampling. The tokens play a crucial role in shaping patterns within tokenized sequences and determining the operations that the language model can comprehend.

Common methods for tokenization, such as Byte Pair Encoding (BPE), often break down a single number into segments that do not correspond directly to the number [28]. For instance, the GPT-3 tagger decomposes the number 42235630 into [422,35,630]. New language models like LLaMA [29] typically treat numbers as a single entity by default. LLMTIME adopts a method where it separates each digit of a number using spaces, tokenizes each digit individually, and uses commas to distinguish between different time steps in a time series. Since decimal points do not add value for a specific precision level, LLMTIME eliminates them during encoding to reduce the length of the context. For example, with a precision of two digits, we preprocess a time series by converting it from

LLMTIME resizes the value to ensure that the rescaled time series value at the $\alpha$-percentile is 1. It refrains from scaling by the maximum value to allow the LLM to observe instances $(1-\alpha)$ where the quantity of numbers varies, and it mimics this pattern in its results to generate a larger value than it observes. Furthermore, LLMTIME endeavors to incorporate an offset $\beta$ computed from the percentile of the input data and fine-tunes these two parameters in accordance with the verification log-likelihood.

Although LLMTIME may perform similarly to the traditional autoregressive model ARIMA on some datasets, its forecasting accuracy is lower than ARIMA when dealing with other datasets such as traditional and artificial signals that exhibit noisy almost periodic patterns. Moreover, as the data values increase over time, the forecasting performance of LLMTIME diminishes noticeably.

In Darts [31], the darts.datasets module offers a variety of pre-existing time series datasets suitable for showcasing, testing, and experimentation purposes. These datasets typically consist of traditional time series data, with Darts specializing in time series forecasting and boasting a comprehensive forecasting system. We utilized LLMTIME and ARIMA models to conduct forecasting on eight datasets (’AirPassengersDataset’, ’AusBeerDataset’, ’GasRateCO2Dataset’, ’MonthlyMilkDataset’, ’SunspotsDataset’, ’WineDataset’, ’WoolyDataset’, ’HeartRateDataset’) integrated into Darts. A portion of the experimental outcomes is depicted in Fig. 1. The AirPassengersDataset comprises classic airline data, documenting the total monthly count of international air passengers from 1949 to 1960, measured in thousands. All experimental findings are detailed in Appendix  V-A.

The experimental results demonstrate that the predictive performance of the LLMTIME approach utilizing GPT-3.5 on the Darts dataset is notably inferior to that of the ARIMA method. It is evident that as the data values increase steadily over time, the forecasting accuracy of LLMTIME diminishes significantly, a pattern observed in other data experiments as well.

The Monash dataset, referenced as [32], comprises 30 openly accessible datasets and baseline metrics for 12 predictive models. There are various versions of the datasets due to differences in frequency and handling of missing values, resulting in a total of 58 dataset variations. Additionally, the dataset collection includes a diverse range of real-world and competition time series datasets across different domains. Similar to the LLMTIME approach, we assessed GPT-3’s zero-shot performance on the top 10 datasets with the most effective baseline model. A noticeable decline in the LLMTIME forecasting performance is evident in the FredMd dataset, which is a monthly database focused on Macroeconomic Research, as described in Working Paper 2015-012B by Michael W. McCracken and Serena Ng. This database aims to provide a comprehensive monthly macroeconomic dataset to facilitate empirical analysis that leverages ”big data.” (The outcomes are depicted in Fig. 2 for FreMD and CovidDeaths, with additional details in Appendix  V-B).

In this section, we examined time series data related to the economy, specifically focusing on the total exports of six countries over the past few decades. The data was sourced from CEIE [35], a comprehensive database offering economic information on over 213 countries and regions, encompassing macroeconomic indicators, industry-specific economics, and specialized data. Figure 3 illustrates the total export value of the United Kingdom from January 1989 to December 2023, denominated in millions of dollars and plotted against months.

A range of artificial signals were forecasted, and data was produced by adding noise to the almost periodic function

where the term $noise$ denotes Gaussian noise with a variance of $\sigma^{2}$. The almost periodic function has traditionally been significant in the development of harmonic analysis, leading to the formulation of Wiener’s general harmonic analysis theory (GHA) [2], and subsequently influencing the study of statistics of random processes.

The function (2) corresponds to an almost periodic function. In the field of mathematics, an almost periodic function is characterized by having multiple frequency components that do not share any common factor. Almost periodicity is a feature of dynamical systems that seem to revisit their trajectories in phase space, albeit not precisely. Several experiments were carried out by varying the standard deviations ($\sigma$). The outcomes of certain experiments ($\sigma$=0.1) are depicted in Fig. 4.

Furthermore, we conducted experiments on the four standard deviations of Gaussian noise using LLLTIME and ARIMA methods, and computed their mean square errors to generate Fig. 5.

The comparison of the mean square errors indicates that the forecasts generated by the LLMTIME model generally exhibit higher errors compared to those produced by the ARMIR model. Additionally, the forecasting effectiveness graphs demonstrate that the ARMIR model outperforms the LLMTIME model significantly. For further details on the remaining experimental outcomes, please refer to Appendix  V-D.

Furthermore, we examined the sine wave and the combination of sinusoidal and linear functions. The findings indicate that LLMTIME is effective only for time series that exhibit complete periodicity, while it struggles to forecast non-periodic signals. Refer to Fig. 6 for more information.

Experimental outcomes obtained from the aforementioned datasets allow us to calculate the Mean Squared Error (MSE) between the predicted outcomes and the actual data. Our analysis indicates that, in most instances, the predictive accuracy of the LLMTIME model is inferior to that of the ARIMA model. Refer to Table I for details.