Scaling Law for Time Series Forecasting

For a time series $s_{0,1,\ldots,L}$ of length $L$ (each $s_{i}$ is a unit element in the sequence that may contain single or multiple variables, dependent or independent), we represent it using a vector $x$ in $\mathcal{M}(L)$.

We make these assumptions on the spaces $\{\mathcal{M}(1),\mathcal{M}(2),\ldots\}$ and the distribution of data in the spaces:

• 1.

  Information-preserving: Intuitively speaking, we should be able to recover the real sequence (which might be a multivariable sequence or a singlevariable sequence) from its corresponding intrinsic vector with the error bounded by a small constant value. Formally we can state this as follows:

  Exists a mapping $\phi$ from the original length-$L$ sequence space $\mathcal{O}(L)$ to $\mathcal{M}(L)$, an inverse mapping $\phi^{-1}:\mathcal{M}(L)\to\mathcal{O}(L)$ and a constant $e\ll 1$ related to $L$ so that for any $\mathbb{E}_{x\sim\mathcal{O}(L)}$, $\|x-\phi^{-1}(\phi(x))\|_{2}^{2}\leq e(L)$.

• 2.

  Inverse Lipschitz: $\phi^{-1}$ should be $K_{I}$-Lipschitz under L2 norm. That is:

  $$\forall x,y\in\mathcal{M}(L),\|\phi^{-1}(x)-\phi^{-1}(y)\|_{2}\leq K_{I}\|x-y\|_{2}$$

• 3.

  Bounded: For simplicity, we assume the values in all dimensions of the intrinsic space are bounded, and thus we can scale the intrinsic space to fit it into $\mathcal{M}(H)=[0,1]^{d_{I}(H)}$.

• 4.

  Isomorphism: $\mathcal{M}(L_{1})$ is isomorphic to a subspace of $\mathcal{M}(L_{2})$ for $L_{1}\leq L_{2}$. Moreover, the isometry should also preserve the data distribution in the space. Formally, we can state it as:

  Let $P[H_{1},H_{2}]$ denote the linear projection from $\mathcal{M}(H_{1})$ to the subspace of it isomorphic to $\mathcal{M}(H_{2})$, and let $Cov_{L}$ denote the covariance matrix of data distribution in $\mathcal{M}(L)$, then $Cov_{L_{1}}$ should be congruent to $P[L_{2},L_{1}](Cov_{L_{2}})$ for any $L_{1}<L_{2}$.

• 5.

  Linear Truncation: Truncation in time series space is close to linear projection mapping in $\mathcal{M}(L)$. Formally, we can state it as:

  Define a truncating function $t_{p}[H_{1},H_{2}](x_{0:H_{1}})$, so that for any sequence $s$, let $s_{h_{1}:h_{2}}$ denote the intrinsic vector for the subsequence from $s[h_{1}]$ to $s[h_{2}-1]$, then $t_{p}[H_{1},H_{2}](s_{0:H_{1}})=s_{0:H_{2}}$, then $t_{p}[H_{1},H_{2}]\approx P[H_{1},H_{2}]$.

• 6.

  Causality: There should be an optimal model to predict the next $S$ frames given the previous $h\to\infty$ frames, so that the error only originates from the intrinsic noise. That is:

  $$\exists F[S]:\mathcal{M}\to\mathcal{M}(S),s.t.\lim\limits_{h\to\infty}\mathbb{P}(y\mid x_{-h:0})=(1-\eta)\delta(F[S](x_{-h:0}))+\eta\mathcal{N}(F[S](x_{-h:0}),\Sigma_{S}),$$

  where $\eta$ stands for the noise ratio in the system, $\delta(\cdot)$ stands for the Dirichlet function and $\mathcal{N}(\mu,\Sigma)$ stands for a normal distribution with mean $\mu$ and covariance $\Sigma$. (Notice that the noise distribution is not necessarily a normal distribution and our result holds for any noise distribution with mean $\mu$ and covariance $\sigma$. However we only consider the normal distribution case here for simplicity.)

  Moreover, we assume $F[S]$ is first-order $K_{1}$-Lipschitz and second-order $K_{2}$-Lipschitz in $L_{2}$ metric (as we assume $S$ is fixed we don’t discuss how the Lipschitz coefficients vary with $S$ and simply take them as constant).

• 7.

  Uniform Sampling Noise: When drawing a length-$L$ sample, we assume the sampling noise is uniform in each direction in $\mathcal{M}(L)$.

• 8.

  Zip-f Distribution: We assume the data distribution in the intrinsic space follows a Zip-f law on different dimensions of the intrinsic space. That is, the eigenvalue spectrum of $Cov_{L}$ satisfies $\lambda_{i}\approx\lambda_{0}i^{-\alpha_{Z}}$ where $\lambda_{i}$ represents the $i$-th largest eigenvalue. This is shown by other work [34, 35] and also verified in our experiments.

This result is asymptotic and does not suggest uniform intrinsic dimensions across sequence elements. Specifically, for a sequence element with $v$ variables, $d_{I}(H)\propto v$ approximately in channel-independent scenarios. In contrast, for channel-dependent cases, which are more common, the total intrinsic dimension is typically less than the sum of dimensions for individual variables.

In the deduction part we do not assume the specific relationship between $d_{I}(H)$ and $H$. Some previous work[36] shows that in some cases $d_{I}(H)\approx\Theta(H)$. In Appendix A.1 we discuss more about their relationship.

We formulate the intrinsic space and these assumptions more strictly and provide a brief construction of $\{\mathcal{M}(1),\mathcal{M}(2),\ldots\}$ in Appendix A.2. Also, we discuss cases where these conditions are not strictly satisfied in Appendix A.3.

Moreover, under these assumptions made, the loss in the original space $L_{ori}$ can be linearly bounded by the loss in the intrinsic space $L_{ins}$, details of which can be found at Appendix A.4.

In the following deduction, we use $L$ to denote $L_{ins}$.

In the following sections, we consider the task to predict a vector in $\mathcal{M}(S)$ given the vector corresponding to its previous $H$ frames in $\mathcal{M}(H)$ with a model $m$. For simplicity, we represent the operation of obtaining a vector in $\mathcal{M}(t_{2}-t_{1})$ by truncating the sequence $s$ from time $t_{1}$ to $t_{2}$ as $x[t_{1}:t_{2}]$, where $x$ is a representation in $\mathcal{M}(|s|)$.

Let $m^{*}$ denote the optimal Bayesian model, then it should satisfy:

Thus:

Since $x[H:H+S]-m^{*}(x[0:H])$ and $m^{*}(x)-m(x)$ are independent with respect to $s[H:H+S]$ and $E_{x\sim\mathcal{M}(H+S)}[x[H:H+S]-m^{*}(x[0:H])]=0$, the loss is a sum of the previous two terms: one is decided by the capability of the optimal Bayesian model (or the Bayesian Error), and the other is the model’s ability to approximate the Bayes Estimation: $L=L_{Bayesian}+L_{approx}$.

We then calculate each of the two terms corresponding to the horizon, dataset size and model size.

We consider the Bayesian error for $\mathcal{M}(\infty)$, when the loss for predicting $S$ frames originates from the inherent uncertainty of the system. That is, we evaluate the amount of information carried by $\mathcal{M}(H)$ compared to $\mathcal{M}(\infty)$. By assumption 5 and 6 in section 3.2.1, It can be verified that:

According to assumption 7, the noise in the $i$-th predicted frame would be proportional to $\sqrt{i}$, and the total noise in the predicted $S$ frames would be proportional to $S$. Let $\sigma_{M}^{2}$ denote the variance of the noise for a single frame in a single dimension, then $tr(\Sigma_{S})$ should be equal to $\sigma_{M}^{2}S^{2}d_{I}(S)$.

According to assumptions 4 and 8, we can express the inverse projection term into:

which indicates that:

The training data is sampled from the distribution of $E_{x\sim\mathcal{M}(H+S)}[x[H:H+S]|x[0:H]]$. Following previous works [16, 37], we utilize the piece-wise linear assumption for deep learning models. That is, we assume the model partitions the intrinsic space into $N$ subregions, and does a linear prediction for each subregion independently. Here we discuss two cases: the large-dataset limit and the small-dataset limit.

If the dataset is large and contains many samples in the piece of the model, the model tends to learn the averaged behavior of the region. Intuitively, a larger dataset size brings smaller noise averages, and a larger model brings smaller piece-wise linear pieces, reducing the error caused by deviation of the data distribution to the linear model in the small blocks. Roughly speaking, the loss consists of two terms: one caused by the uncertainly within the subregions partitioned by the model $L_{r}$, and one caused by the noise in the data that makes the model fail to learn the optimal model $L_{n}$. If we assume that the model partitions the space uniformly, then the loss should satisfy:

The uniform partitioning should be the most naive case for the model to learn, and we also analyze on the cases where more advanced partitionings are learned in Appendix D. Also, it is worth noticing that the noise consists of two term, one is the systematic uncertainly as shown in section 3.2.3, and the other is caused by the horizon limitation so that the effect of unseen dimension seems exactly like noise for the model. Please refer to Appendix B.2 for a more detailed derivation.

Otherwise if there are few data samples, or the model is sufficiently large and the model cannot learn to average the noises of closed samples, but rather learns to remember each sample in certain pieces. This would give a data-scaling loss determined by nearest-neighbor distance. In this case:

Please refer to Appendix B.3 for a more detailed derivation of this scenario.

Boundary of the two phases is decided by Dataset size and Model size, as well as horizon. It is worth noticing that in time series forecasting tasks, the sliding window method for constructing data samples is usually used. ³³3Data points between time [t-H:t+S] is considered one sample, and [t-H+1:t+S+1] is considered another sample in the dataset. These two samples are strongly correlated with each other. Therefore, the closest data points are always dependent on each other and are strongly correlated. The effective number of mutually independent data fragments is approximately proportional to $D/H$ rather than $D$, and $\xi=D/NH$ would be a very approximate order parameter separating the previous two scenarios. Again, please refer to Appendix B for a more detailed derivation.

For the case with sufficient data, the total loss is:

We consider two cases. The detailed derivation can be found in Appendix C.1.

If model size is too small compared to dataset size such that $N=o(D^{\frac{d_{I}(H)}{d_{I}(H)+4}})$, then the effect of dataset size on picking optimal horizon could be neglected, thus:

where $\mathcal{W}(\cdot)$ is the Lambert W function ($\mathcal{W}(x)\approx x$) and $C_{0}=\frac{K_{1}^{2}\pi^{2}(1-\eta)\lambda_{0}}{K_{2}^{2}}$.

If N is not that small: $N=\omega(D^{\frac{d_{I}(H)}{d_{I}(H)+4}})$. Then the noise effect would be dominant in picking optimal $H$, and the optimal $d_{I}(H)$ would be:

In this case, $d_{I}^{*}$ grows noticeably with the increment of $D$ and the decrement of $N$.

If data is scarce, the loss could be written as:

It is worth noticing that in this case, we assume the model is always large enough to find the nearest neighbor for a test sample, hence the loss is irrelevant with $N$. We can estimate the optimal $d_{I}^{*}$ as:

Where $C_{s}$ is a constant irrelevant with $D$, $N$ or $H$. The exact form is provided in Appendix C.2. Compared to the first scenario, the optimal $d_{I}$ changes much less in this scenario.

Optimal Horizon grows with more available training data. We conduct experiments, fixing the available training data amount and model size while adjusting the horizon. As shown in Figure 3, an optimal horizon is observed at each data amount, and this optimal value grows with more available training data.

A dataset with a larger feature degradation has a smaller optimal horizon. As our theory predicts, larger $\alpha_{z}$ leads to a smaller optimal horizon. In the experiment in Figure 4, we use the same Linear model on two different datasets with comparable amount of training data: the Exchange Dataset has $70\%$ available training data compared to the ETTh1 Dataset. We use eigenvalues obtained by doing Principal component analysis on sampled series as an approximation to the feature variance in the intrinsic space. Detailed procedure and more PCA results can be found at Appendix G.

Channel-Dependent and Channel-Independent Models sometimes are in different states of the two cases. For CD models, $d_{I}(H)$ is larger and less training data is available, hence it tends to be in the few-data limit. For CI models, $d_{I}(H)$ is smaller and $D$ is larger, hence it may reach the data-dense limit (where the scaling exponent for $D$ is $-1$).

In the following Figure 5, iTransformer on Traffic dataset is in the data-dense limit. For MLP on Transformer, when the horizon is small it is in the few-data limit. For the Linear model, since it is simply linear (rather than piece-wise linear), we expect it to be within the data-dense limit even for long horizons and when the dataset is relatively small (like ETTh1).

The advantage of channel-independent and channel-dependent models can be explained from the perspective of our theory. For channel-dependent models, the horizon limitation is smaller hence it performs better with plenty of training data. For channel-independent models, more training data is available; moreover, $d$ is smaller, making the scaling exponent larger.

As predicted by our theory there are two cases. (1) When $N$ is small, the optimal horizon does not change much with $N$. (2) When $N$ is large, the model size scaling term no longer dominates, the coefficient of the noise term $\frac{Nd_{I}}{D}$ dominants thus larger $N$ leads to smaller optimal $H$.

From the observations in Figure 6, it can be discerned that in the initial scenario, where an enhancement in N results in a performance augmentation, the optimal horizon remains relatively invariant with respect to $N$. This is exemplified in the case of the SingleLayerMLP model. However, in the second scenario, where N attains its optimal value (i.e., for a certain horizon, a smaller N surpasses a larger $N$ in performance), a larger $N$ will correspond to a reduced optimal horizon. This phenomenon is evident in the instances involving the ModernTCN model.

As predicted by our theory, we see that dataset size has an impact on optimal horizon, while model size has a less significant influence on it.