DSformer: A Double Sampling Transformer for Multivariate Time Series Long-term Prediction

At present, deep learning has been widely studied in the field of multivariate time series long-term prediction (Tan et al., 2022). As one of the most classical deep learning algorithms in time series prediction, recurrent Neural Network (RNN) (Liu et al., 2021a) has been widely studied. As the most classical variant of RNN, the long short-term memory network (LSTM) (Liu et al., 2020) and the gated recurrent unit (GRU) (Dong et al., 2021) have made progress in the field of time series prediction. Compared with RNN, LSTM and GRU effectively solve the gradient problem and improve their prediction accuracy (Liang et al., 2023a). In addition to RNN-based models, the convolutional neural network (CNN) (Mi et al., 2022) based models have also been proven to have effects in the field of multivariate time series long-term prediction. For example, Temporal Convolutional Network (TCN) (Yu et al., 2023) improves the ability of sequence modeling by introducing Dilated Causal Convolutions and residual connections. Besides, with the improvement of computer performance, the idea of fusing different network structures is constantly proposed (Yan et al., 2022). LSTMa (Zhou et al., 2023) improved the ability of the model to mine temporal information by combining LSTM and attention mechanism. Besides, by effectively combining LSTM, CNN and attention mechanism, LSTNet (Lai et al., 2018) achieved better results than traditional methods in multivariate time series long-term prediction. However, the above models have limitations in mining the key context information of long sequence and the correlation of different variables, which limits their performance.

Due to its excellent series modeling capabilities, Transformer variants have seen rapid growth in multivariate time series long-term prediction (Zerveas et al., 2021). Li et al. (Li et al., 2019) used the convolutional self-attention mechanism to improve the sequence modeling ability of the traditional transformer and proposed the LogSparse transformer (LogTrans). Kitaev et al. (Kitaev et al., 2019) combined Locality sensitive hashing attention with reversible residual layers to improve the ability to analyze long-term dependencies and proposed Reformer. Zhou et al. (Zhou et al., 2021) proposed Informer by introducing a ProbSparse self-attention mechanism and the generative decoder. Liu et al. (Liu et al., 2022b) proposed Pyraformer, which introduces the pyramidal attention module and multi-resolution modeling approach. The above models focus on optimizing the ability of attention to analyze the long-term dependence, but they do not fully analyze the characteristics of time series. Different from the above methods, Autoformer (Wu et al., 2021a) introduces autoregressive attention and deep decomposition structure to realize long-term prediction of time series. The deep learning decomposition structure improves the ability of the model to analyze trends and seasons. On this basis, FEDformer (Zhou et al., 2022b) and TDformer (Zhang et al., 2022a) introduce the deep decomposition framework and Fourier Attention to realize the long-term prediction of time series. These methods improve the ability to mine time series context relation by introducing trend and seasonal modeling. However, the decomposition method transform raw data into fixed forms based on expert experience, which limits the ability of the model to mine the data itself. At present, to strengthen the model’s ability to mine global information from raw data, Patch TST (Nie et al., 2023) and Crossformer (Zhang and Yan, 2023) adopt the idea of patch modeling. In addition, Patch TST and Crossformer respectively adopt channel independence and two-stage attention to realize multivariable modeling. Due to the mining of more data features, Patch TST and Crossformer can achieve better capabilities than the transformer variants mentioned above. However, Patch TST ignores the correlation between different variables, and Crossformer ignores the role of local information. In general, the existing models do not make full use of the key features of multivariate long sequence time series, which limits their performance.

In this section, we introduce the basic definition of multivariate time series and multivariate time series long-term prediction.

The overall framework of the DSformer is given in Figure 2. And it can be found that DSformer contains two important component: the double sampling block and the temporal variable attention block. And DSformer combines a double sampling block and three temporal variable attention blocks to mine three features and fully perform information integration. In this section, we intuitively discuss each block of DSformer and its parallel structure.

Firstly, we discuss the DS block, which uses the downsampling and the piecewise sampling to process the original input features respectively. The downsampling converts the original sequence into multiple subsequences with simliar length by increasing the sampling interval. The global information of subsequences with larger time intervals is more significant than that of the original sequence (Mostafa et al., 2022). The piecewise sampling can divide the long sequence into multiple contiguous fragments. Because the observation length of the time series is reduced, the model can mine the local information more intensively (Zhang et al., 2022b). At the same time, to reduce the information loss caused by sampling, we connect the subsequences obtained from the sampling method and convert them into 3D tensors.

Second, the TVA block aims to mine the 3D tensors processed by the DS block from the temproal dimension and the variable dimension. Based on the parallel structure, the TVA block enable the temporal attention and the variable attention to mine input features respectively. Different from the traditional stacked multi-layer structure, the parallel structure enables the model to mine information more centrally (Fu et al., 2022b). Then the effective integration of temporal information and variable information is realized through addition and layer normalization.

Finally, the overall framework of DSformer also adopts parallel structure to realize feature mining and modeling. Specifically, the two different 3D tensors obtained by the DS block are mined by two TVA blocks. And then, a new TVA block is used to achieve the fusion of above two processed tensors. Therefore, DSformer can be used to mine global information, local information and variable correlation in parallel. Based on this structure, DSformer can strengthen the ability of mining features and achieving fusion.

The double sampling block consists of two important steps: the down sampling and the piecewise sampling. Figure 3 presents a schematic of these two sampling methods. These two sampling methods transform the original 2D feature vectors $X\in R^{N*H}$ into two 3D features $X_{ds}\in R^{N*C*\frac{H}{C}}$ and $X_{ps}\in R^{N*C*\frac{H}{C}}$. The feature vector obtained by downsampling contain more global information. The feature vector obtained by piecewise sampling contains more local information. In the following, we briefly describe the proposed two sampling methods.

Down sampling. For a time series with length $H$, we obtain $C$ subsequences of consistent length in the same way as shown in Figure 3 (a). The subsequence obtained by the downsampling method has a larger time interval. As a special form of sequence, observing time series data with larger time intervals can obtain more intuitive global information. In addition, to avoid the information loss caused by down-sampling, we put $C$ subsequences together and obtain the feature vector $X_{ds}\in R^{N*C*\frac{H}{C}}$ for subsequent modeling. For the $jth$ subsequence, its main constituent form is given as follows:

Piecewise sampling. For a time series with length $H$, we obtain $C$ subsequences of consistent length in the same way as shown in Figure 3 (b). The piecewise sampling method can transform the original time series into continuous subsequence with the same length. Each subsequence contains local information over a historical period of time. Unlike down sampling, piecewise sampling allows the model to focus more attention on local information, which usually reflects the details of local changes over a cycle. In addition, to avoid the information loss caused by piecewise sampling, we put $C$ subsequences together and obtain the feature vector $X_{ps}\in R^{N*C*\frac{H}{C}}$ for subsequent modeling. For the $jth$ subsequence, its main constituent form is given as follows:

After obtaining two different feature vectors $X_{ds}\in R^{N*C*\frac{H}{C}}$ and $X_{ps}\in R^{N*C*\frac{H}{C}}$ by the DS block, we next introduce how to use TVA block to mine above feature vectors from temporal dimension and variable dimension.

The proposed TVA block consists of two main components: temporal attention and variable attention. The main function of temporal attention is to mine the context information of data from the temporal dimension. The main function of variable attention is to mine the internal implicit relation between different variables. The information mined by these two components is integrated through a parallel structure. Figure 4 illustrates the detailed composition of TVA block, temporal attention and variable attention. Next, we present the modeling details of temporal attention, variable attention, and TVA blocks.

For the $X_{ds}\in R^{N*C*\frac{H}{C}}$ and $X_{ps}\in R^{N*C*\frac{H}{C}}$ obtained by the double sampling block, they are transmitted to the temporal attention and variable attention as input to the TVA block. Then, temporal attention and variable attention will process above two feature vectors in the following way:

Temporal attention. Temporal attention consists of three main components (multi-head attention, residual connection, and layer normalization).

Firstly, multi-head attention is used to mine the time dimension of the input feature vector $X_{ds}\in R^{N*C*\frac{H}{C}}$ and obtain the processed feature vector ${X^{ta}}_{ds}\in R^{N*C*\frac{H}{C}}$.

where, $FC(.)$ stands for the fully connected layer. $softmax(.)$ stands for the normalized exponential function. $K^{T}$ stands for $K$ after converting the last two dimensions ($K^{T}\in R^{N*\frac{H}{C}*C}$).

Then, the output ${X^{TA}}_{ds}\in R^{N*C*\frac{H}{C}}$ of the temporal attention component is obtained by the residual connection and the layer normalization:

where, $U$ and $\sigma$ are represent the statistics of the feature vectors. $g$ is the gain. $b$ is the bias. $\sigma$ is a small decimal number that prevents division by zero.

Variable attention. Different from temporal attention, variable attention mainly uses multi-head attention to mine data from the perspective of the number $N$ of variables. Through the mining of variable attention, DSformer can effectively analyze the correlation between different variables and conduct information interaction. The formulas of the variable attention are given as follows:

where, $d_{k}$ can let the outcome of $Q*K^{T}$ satisfy the distribution with expectation 0 and variance 1. In particular, in the above formula, the corresponding matrix forms of $Q$ and $K^{T}$ are $\frac{H}{C}*C*N$ and $\frac{H}{C}*N*C$, respectively

Based on above temporal attention method and variable attention method, ${X^{TA}}_{ds}\in R^{N*C*\frac{H}{C}}$ and ${X^{VA}}_{ds}\in R^{N*C*\frac{H}{C}}$ are obtained. Then, ${X^{TA}}_{ds}$ and ${X^{VA}}_{ds}$ are integrated and the output ${X^{{}^{\prime}}}_{ds}\in R^{N*\frac{H}{C}}$ of TVA block is obtained by the following formula:

where, $F_{LN}(.)$ stands for layer normalization. In addition, the main function of $FC(.)$ is to transform the feature vector dimension from $N*C*\frac{H}{C}$ to $N*\frac{H}{C}$.

Information integration based on TVA block. The feature vector $X_{ps}\in R^{N*C*\frac{H}{C}}$ is mined in the same way as above methods. And the output feature vector ${X^{{}^{\prime}}}_{ps}\in R^{N*\frac{H}{C}}$ is obtained. For ${X^{{}^{\prime}}}_{ps}$ and ${X^{{}^{\prime}}}_{ds}$, we first used the layer normalization to achieve preliminary information fusion.

Then, the two-dimensional feature vector $X^{{}^{\prime}}\in R^{N*\frac{H}{C}}$ was used as the input to the TVA block and fully mined from the temporal dimension and variable dimension.

Different from the previous modeling form, the feature vectors ${X^{{}^{\prime}}}_{ps}\in R^{N*\frac{H}{C}}$ processed by the the information fusion method based on TVA block are 2D tensors. Therefore, the main matrix forms of the variables $Q$ and $K^{T}$ (temporal attention) modeled here are $N*\frac{H}{C}$ and $\frac{H}{C}*N$ respectively.

Finally, the feature vectors, which are further mined and integrated by TVA block, are passed to the multi-layer perceptron to effectively realize the output of the final prediction result $Y\in R^{N*L}$.

DSformer is constructed by effectively combining the double sampling block and three TVA block. The double sampling block effectively obtains the feature vectors containing key information. TVA block mines different feature vectors and fully realizes information integration. The specific modeling steps of the proposed DSformer are given as follows:

Step I: For the original 2D input features $X\in R^{N*H}$, the data is transformed into two 3D features $X_{ds}\in R^{N*C*\frac{H}{C}}$ and $X_{ps}\in R^{N*C*\frac{H}{C}}$ by a double sampling block.

Step II: Two TVA blocks are used to model and analyze $X_{ds}$ and $X_{ps}$, respectively. TVA block deeply mines feature vectors from both temporal dimension and variable dimension. In addition to this, the 3D features are transformed into 2D features ${X^{{}^{\prime}}}_{ds}\in R^{N*\frac{H}{C}}$ and ${X^{{}^{\prime}}}_{ps}\in R^{N*\frac{H}{C}}$ by the feedforward neural network.

Step III: Add ${X^{{}^{\prime}}}_{ds}$ and ${X^{{}^{\prime}}}_{ps}$. And then layer normalization is used to process the new feature vector $X^{{}^{\prime}}\in R^{N*\frac{H}{C}}$.

Step IV: The TVA block is used to further mine the feature vector $X^{{}^{\prime}}$ from the temporal dimension and the variable dimension. At the same time, the mined feature vectors are passed to the MLP for long-term prediction.

Step V: Based on the MLP for decoding, the model finally obtains the prediction result $Y\in R^{N*L}$ with the prediction step of $L$. The decoding process is calculated using the following formula:

where, $X^{{}^{\prime\prime}}$ stands for the feature vector obtained after the TVA block processing in step IV.

Based on the above steps, DSformer can effectively analyze and mine the key features and obtain the multivariate time series long-term prediction results. In addition, to ensure the training effect of the model, we adopt the method of fusing L1 Loss and L2 Loss for error backpropagation. The formula is given as follows:

where, $Y$ represents the prediction result of the model. $Y^{tru}$ stands for the true label. $B$ represents the number of samples. $N$ stands for the number of variables. $L$ stands for prediction step size. $w_{L1}$ represents the weight of Loss.

Dataset. In order to fully verify the effectiveness of the proposed DSformer in the field of multivariate time series long-term prediction, this paper selects nine classical data sets for comparative experiments. These datasets include ETT (ETTh1, ETTh2, ETTm1 and ETTm2), Exchange, ILI, Weather, Electricity and Traffic (Zhou et al., 2022b). Table 1 presents the basic statistics of these datasets.

Table 3 shows the prediction results of the proposed DSformer and all baselines on nine datasets. The best results are highlighted in bold and the second best results are underlined. Based on Table 3, the following conclusions can be obtained: (1) Compared with other SOTA methods, Informer and Pyraformer have larger prediction errors. Although these two methods design advanced attention structures to improve the performance of the model, they fail to fully mine the core features of time series. (2) Autoformer and FEDformer improve their prediction performance by introducing trend-season decomposition. However, the decomposition method converts the original sequence into a fixed form based on expert experience, which limits the ability of the model to mine the original data. (3) Compared with transformer variants mentioned above, Patch TST, TimesNet and Crossformer focus on mining global information and variable correlation from original data, which enables them to achieve better prediction results. However, for multivariate long sequence time series, they do not make full use of the three features proposed in this paper, which makes their performance limited. (4) Compared with the existing SOTA models, the DSformer can achieve satisfactory prediction results. Firstly, two feature vectors focusing on global information and local information can be obtained through the DS block. Then, TVA block can mine and model these feature vectors from temporal dimension and variable dimension. Finally, DSformer uses a parallel structure to integrate the above feature information and realize long-term prediction. Therefore, the DSformer can achieve better performance than other SOTA methods.

The DSformer contains four key components: piecewise sampling, down sampling, temporal attention and variable attention. To prove that these components can help the DSformer to mine the key feature information, wu conducts ablation experiments from the following five perspectives: (1) wo/ ps: the piecewise sampling component is removed. (2) wo/ ds: the down-sampling component is removed. (3) wo/ as: In this part, we remove the double sampling block. (4) wo/ ta: we remove temporal attention and replace it with multi-layer perceptron. (5) wo/ va: variable attention is removed.

Figure 5 illustrates the results of the ablation experiments. Based on the experimental results, we can obtain the following conclusions: (1) When there is a correlation between different variables, deleting the variable attention will increase the prediction error of the model. According to the experimental results, the correlation between different time series in Weather data set is large, so the variable attention have a great influence on the prediction results. (2) After deleting the temporal attention, the prediction performance of DSformer decreases significantly. The main reason is that the most important step in time series modeling is mining the time series context relation. If temporal attention is lost, it is difficult for DSformer to effectively analysis the context relation of time series data. (3) When the prediction step size is long, removing down sampling significantly increases the error of the prediction. When the prediction step size is short, deleting piecewise sampling significantly increases the prediction error. The main reason is that the features obtained by down sampling contain more global information, and the features obtained by interval sampling contain more local information. Therefore, they will affect the modeling effect of different prediction steps, respectively. (4) After removing the down-sampling and piecewise sampling at the same time, the prediction error of the DSformer further increases. The main reason is that these two sampling methods can deepen the model’s ability to focus on learning the global and the local respectively. When the sampling method is removed, the model may not be able to focus on the key information, which increases the prediction errors of the DSformer. (5) The results of ablation experiments can demonstrate the importance of the proposed four components, which can effectively mine the three main features of multivariate long sequence time series and reduce prediction error.

The hyperparameters of the DSformer usually affect the final prediction results. In order to fully analyze the sensitivity of the DSformer and the role of some key hyperparameters, this section conducts an experimental analysis of the four main hyperparameters (learning rate, number of multi-head attention, sampling interval and weight of Loss) on ETTm2 dataset. Figure 6 illustrates the results of the hyperparameter analysis experiments.

Based on the experimental results, we can get the following conclusions: (1) The number of multi-head attention and the weight of Loss have relatively little impact on the prediction performance of the model. For multi-head attention, an appropriate number can prevent overfitting while ensuring modeling performance. For the weight of Loss, an appropriate value can ensure the training effect of the model and improve the overall performance. (2) The learning rate has a large impact on the model performance. The main reasons include the following two aspects: On the one hand, a large learning rate will produce phenomena such as overfitting. On the other hand, a smaller learning rate will affect the effect of training. Therefore, the setting of learning rate is very important to ensure the training effect of DSformer. (3) As one of the main hyperparameters, the sampling interval has a relatively large impact on the prediction results. When the sampling interval is small, DSformer can achieve better results with shorter prediction steps. When the sampling interval is relatively large, DSformer can achieve better results on longer prediction steps. However, when the sampling interval is too large, the prediction error of DSformer increases significantly. The main reason is that too large sampling interval makes the model lose more local information, which resulted in insufficient usage of information and reduced prediction accuracy. Therefore, setting the sampling interval appropriately can affect the effect of different prediction steps of the DSformer.(4) The sampling interval $C$ of the double sampling block is an important parameter because it affects the input feature information of the DSformer. In the next section, we will future analyze the influence of sampling interval $C$ and input history length $H$ on the experimental results.

The history length affects the information obtained by the model. And the sampling interval affects the model’s utilization of information. Considering the effect of the history length and the sampling interval on the input information of DSformer, we further analyze the influence of these two hyperparameters on the prediction results in this section. To adequately evaluate these two key hyperparameters, we carried out the following comparison experiments: (1) Based on the hyperparameter experiments, it can be found that too large sampling interval is not conducive to the experimental results. Therefore, we set the sampling interval of the double sampling block, that is, the number of subsequences $C=[2,3,4,6]$. (2) Considering the characteristics of the prediction step size of the model, the history length of the model is set as $H=[96,192,336]$, respectively. (3) All above parameters were used to construct experiments on the ETTh2 dataset. And the future length of the DSformer is set to $L=[96,192,336,720]$.

Table 4 shows the experimental results for different history lengths and sampling intervals. Based on the experimental results, the following conclusions can be drawn: (1) When the history length is short, the sampling interval $C$ cannot be too large. If the sampling interval is large, the prediction performance of the model will degrade significantly. The main reason is that the increasing sampling interval limits the ability of DSformer to mine the local information of raw data. The loss of too much local information is not conducive to the short-term prediction effect of the DSformer. (2) When the history length is large, increasing the sampling interval $C$ can improve the prediction performance of the model. On the one hand, increasing the sampling interval can make the feature vector obtained by down-sampling contain more global information. On the other hand, when the history length is large, the model contains more historical period information, and increasing the sampling interval can make the piecewise sampling obtain feature vectors that pay more attention to local information. (3) The history length $H$ of DSformer can affect the amount of global and local information obtained by the model. The size of the sampling interval $C$ can make the model focus on different informations of input features. Therefore, the appropriate balance between the history length and the sampling interval can make the model effectively use the overall information and the global information, which can improve the prediction accuracy of DSformer.

In this section, based on the Electricity data sets, we compare the efficiency of the DSformer and other baselines (Dlinear, Pyraformer, Crossformer, FEDformer and Autoformer). Besides, to make a fair comparison, we compare the mean training time of each epoch of several models. The experimental equipment is the Intel(R) Xeon(R) Gold 5217 CPU @ 3.00GHz, 128G RAM computing server with RTX 3090 graphics card. The batch size is set to 16.

Based on Figure 7, it can be found that although the computational complexity of DSformer is $O(N^{2})$, the actual computational resource consumption of DSformer is not large. Specifically, most existing transformer variants use various theoretical methods to reduce computational complexity, but their actual computational resource consumption is not low due to the introduction of many tricks. Compared with above models, DSformer has two advantages: On the one hand, DSformer uses the DS block to reduce the length of the sequence that needs to be modeled. On the other hand, DSformer does not use some tricks that significantly increase computational resource consumption, such as embeding. Therefore, the results of efficiency comparison further prove the practical application value of DSformer.