FrAug: Frequency Domain Augmentation for Time Series Forecasting

Given the data points in a look-back window $x=\{x^{t}_{1},...,x^{t}_{C}\}{{}_{t=1}^{L}}$ of multivariate time series, where $L$ is the look-back window size, C is the number of variates, and $x^{t}_{i}$ is the value of the $i_{th}$ variate at the $t_{th}$ time step. The TSF task is to predict the horizon $\hat{x}=\{\hat{x}^{t}_{1},...,\hat{x}^{t}_{C}\}{{}_{t=L+1}^{L+T}}$, the values of all variates at future T time steps. When the forecasting horizon $T$ is large, it is referred to as a long-term forecasting problem, which has attracted lots of attention in recent research [?; ?].

Deep learning models have dominated the TSF field in recent years, achieving unparalleled forecasting accuracy for many problems, thanks to their ability to capture complicated temporal relationships of the time series by fitting on a large amount of data. However, in many real applications, time series forecasting remains challenging for two main reasons.

First, the training data can be scarce. Unlike computer vision or natural language processing tasks, time series data are relatively difficult to scale. Generally speaking, for a specific application, we can only collect real data when the application is up and running. Data from other sources are often not useful (even if they are the same kinds of data), as the underlining dynamic systems that generate the time series data could be vastly different. In particular, lack of data is a serious concern in cold-start forecasting and long-term forecasting. In cold-start forecasting, e.g., sales prediction for new products, we have no or very little historical data for model training. In long-term forecasting tasks, the look-back window and forecasting horizon are large, requiring more training data for effective learning. Over-fitting caused by data scarcity is observed in many state-of-the-art models training on several benchmark time-series datasets.

Second, some time-series data exhibits strong distribution shifts over time. For example, promotions of products can increase the sales of a product and new economic policies would significantly affect financial data. When such a distribution shift happens, the model trained on previous historical data cannot work well. To adapt to a new distribution, a common choice is re-training the model on data with the new distribution. However, data under the new distribution is scarce and takes time to collect.

Various time series DA techniques are proposed in the literature. For the time series classification task, many works regard the series as a waveform image and borrow augmentation methods from the CV field, e.g., window cropping [?], window flipping [?], and Gaussian noise injection [?]. There are also DA methods that take advantage of specific time series properties, e.g., window warping [?], surrogate series [?; ?], and time-frequency feature augmentation [?; ?; ?; ?]. For the time series AD task, window cropping and window flipping are also often used. In addition, label expansion and amplitude/phase perturbations are introduced in [?].

With the above, one may consider directly applying these DA methods to the forecasting task to expand training data. We perform such experiments with several popular TSF models, and the results are shown in Table 1. As can be observed, these DA techniques tend to generate reduced forecasting accuracy. The reason is rather simple: most of these DA methods introduce perturbation in the time domain (e.g., noise or zero masking), and the augmented data-label pairs do not preserve the semantics consistency required by the TSF task.

There are also a few DA techniques for the forecasting task presented in the literature. [?] proposes DATSING, a transfer learning-based framework that leverages cross-domain time series latent representations to augment target domain forecasting. [?] introduces two DA methods for forecasting: (i). Average selected with distance (ASD), which generates augmented time series using the weighted sum of multiple time series [?], and the weights are determined by the dynamic time warping (DTW) distance; (ii). Moving block bootstrapping (MBB) generates augmented data by manipulating the residual part of the time series after STL decomposition [?] and recombining it with the other series. It is worth noting that, in these works, data augmentation is not the focus of their proposed framework, and the design choices for their augmentation strategies are not thoroughly discussed.

The forecastable behavior of time series data is usually driven by some periodical events. For example, the hourly sampled power consumption of a house is closely related to the periodical behavior of the householder. His/her daily routine (e.g., out for work during the day and activities at night) would introduce a daily periodicity, his/her routines between weekdays and weekends would introduce a weekly periodicity, while the yearly atmosphere temperature change would introduce an annual periodicity. Such periodical events can be easily decoupled in the frequency domain and manipulated independently.

Motivated by the above, we propose to perform frequency domain augmentation for the TSF task. By identifying and manipulating events in the frequency domain for data points in both the look-back window and forecasting horizon, the resulting augmented data-label pair would largely conform to the behavior of the underlying system.

To ensure the semantic consistency of augmented data-label pairs in forecasting, we add frequency domain perturbations on the concatenated time series of the look-back window and target horizon, with the help of the Fast Fourier transform (FFT) as introduced in Appendix. We only apply FrAug during the training stage and use original test samples for testing.

As shown in Figure 1, in the training stage, given a training sample (data points in the look-back window and the forecasting horizon), FrAug (i) concatenates the two parts, (ii) performs frequency domain augmentations, and (iii) splits the concatenated sequence back into lookback window and target horizon in the time domain. The augmentation result of an example time series training sample is shown in Figure 2.

We propose two simple yet effective augmentation methods under FrAug framework, namely Frequency Masking and Frequency Mixing. Specifically, frequency masking randomly masks some frequency components, while frequency mixing exchanges some frequency components of two training samples in the dataset.

Frequency masking: The pipeline of frequency masking is shown in Figure 3(a). For a training sample comprised of data points in the look-back window $x_{t-b:t}$ and the forecasting horizon $x_{t+1:t+h}$, we first concatenate them in the time domain as $s=x_{t-b:t+h}$ and apply real FFT to calculate the frequency domain representation $S$, which is a tensor composed of the complex number. Next, we randomly mask a portion of this complex tensor $S$ as zero and get $\tilde{S}$. Finally, we apply inverse real FFT to project the augmented frequency domain representation back to the time domain $\tilde{s}=\tilde{x}_{t-b:t+h}$. The detailed procedure is shown in Algorithm 1.

Frequency Masking corresponds to removing some events in the underlying system. For example, considering the household power consumption time series, removing the weekly frequency components would create an augmented time series that belongs to a house owner that has similar activities on the weekdays and weekends.

Frequency mixing: The pipeline of frequency mixing is shown in Figure 3, wherein we randomly replace the frequency components in one training sample with the same frequency components of another training sample in the dataset. The details of this procedure are presented in Algorithm 2.

Similarly, frequency mixing can be viewed as exchanging events between two samples. For the earlier example on household power consumption, the augmented time series could be one owner’s weekly routine replaced by another’s, and hence the augmented data-label pair largely preserves semantical consistency for forecasting.

Note that, frequency masking and frequency mixing only utilize information from the original dataset, thereby avoiding the introduction of unexpected noises compared to those dataset expansion techniques based on synthetic generation [?; ?]. Moreover, as the number of combinations of training samples and their frequencies components in a dataset is extremely large, FrAug can generate nearly infinite reasonable samples.

Dataset. All datasets used in our experiments are widely-used and publicly available real-world datasets, including Exchange-Rate [?], Traffic, Electricity, Weather, ILI, ETT [?]. We summarize the characteristics of these datasets in Table 2.

Baselines. We compare FrAug, including Frequency Masking (FreqMask) and Frequency Mixing (FreqMix), with existing time-series augmentation techniques for forecasting, including ASD [?; ?] and MBB [?; ?].

Deep Models. In the main paper, we include four state-of-the-art deep learning models for long-term forecasting, including Informer [?], Autoformer [?], FEDformer [?] and DLinear [?]. In the Appendix, we also provide experiments on LightTs [?], Film [?], and SCINet [?]. The effectiveness of augmentations methods is evaluated by comparing the performance of the same model trained with different augmentations.

Evaluation metrics. Following previous works [?; ?; ?; ?; ?; ?; ?], we use Mean Squared Error (MSE) as the core metrics to compare forecasting performance.

Implementation. FreqMask and FreqMix only have one hyper-parameter, which is the mask rate/mix rate, respectively. In our experiments, we only consider 0.1, 0.2,0.3,0.4,0.5. We use cross-validation to select this hyper-parameter. More details are presented in the Appendix.

Table 3 and Table 4 show the results of long-term forecasting. We use different augmentation methods to double the size of the training dataset. As can be observed, FreqMask and FreqMix improve the performance of the original model in most cases. However, the performances of ASD and MBB are often inferior to the original model.

Notably, FreqMask improves DLinear’s performance by 16% in ETTh2 when the predicted length is 192, and it improves FEDformer’s performance by 28% and Informer’s performance by 56% for the Weather dataset when the predicted length is 96. Similarly, FreqMix improves the performance of Autoformer by 27% for ETTm2 with a predicted length of 96 and the performance of Informer by 35% for ETT2 with a predicted length of 720. These results indicate that FrAug is an effective DA solution for long-term forecasting, significantly boosting SOTA performance in many cases.

We attribute the performance improvements brought by FrAug to its ability to alleviate the over-fitting issues. Generally speaking, a large gap between the training loss and the test loss, the so-called generalization gap, is an indicator of over-fitting. Figure 4 demonstrates training loss and test error curves from deep models Autoformer and Informer. Without FrAug, the training loss of Autoformer and Informer decease with more training epochs, but the test errors increase. In contrast, when FrAug is applied to include more training samples, the test loss can decrease steadily until it is stable. This result clearly shows the benefits of FrAug.

In Figure 5, we visualize some prediction results with/without FrAug. Without FrAug, the model can hardly capture the scale of data, and the predictions show little correlation to the look-back window, i.e, there is a large gap between the last value of the look-back window and the first value of the predicted horizon. This indicates that the models are over-fitting. In contrast, with FrAug, the prediction is much more reasonable.

Another important application of FrAug is cold-start forecasting, where only very few training samples are available.

To simulate this scenario, we reduce the number of training samples of each dataset to the last 1% of the original size. For example, we only use the 8366^th-8449^th training samples (83 in total) of the ETTh2 dataset for training. Then, we use FrAug to generate augmented data to enrich the dataset. In this experiment, we only consider two extreme cases: enlarging the dataset size to 2x or 50x, and the presented result is selected by cross-validation. Models are evaluated on the same test dataset as the original long-term forecasting tasks.

Table 5 shows the results of two forecasting models: DLinear and Autoformer on 4 datasets. The other results are shown in Appendix. We also include the results of models trained with the full dataset for comparison. As can be observed, FrAug maintains the overall performances of the models to a large extent. For the Traffic dataset, compared with the one trained on full training data, the overall performance drop of DLinear is 13% with FrAug, while the drop is 45% without FrAug. Surprisingly, sometimes the model performance is even better than those trained with the full dataset. For example, the performance of Autoformer is 5.4% better than the one trained with the full dataset for ETTh2 when the predicted length is 96. The performance of DLinear is 20% better than the one trained with the full dataset for Exchange Rate when the predicted length is 720. We attribute it to the distribution shift in the full training dataset, which could deteriorate the model performance.

Distribution shifts often exist between the training data and the test data. For instance, as shown in Figure 7, the distribution of data (trend and mean value) changes dramatically at the middle and the end of the ILI dataset.

As time-series data is continuously generated in real applications, a natural solution to tackle this problem is to update the model when the new data is available for training [?]. To simulate the real re-training scenario, we design a simple test-time training policy, where we divide the dataset into several parts according to their arrival times. The model is retrained after all the data points in each part are revealed. Specifically, the model is trained with the data in the first part and tested on data in the second part. Then, the model will be re-trained with data in both the first and second parts and tested on data in the third part.

In our experiments, we divide the dataset into 20 parts and use the above setting as the baseline. Then, we validate the effectiveness of FrAug under the same setting. FrAug can help the model fit into a new distribution by creating augmented samples with the new distribution. Specifically, for data that are just available for training, which is more likely to have the same distribution as future data, we create 5 augmented samples. The number of augmented samples gradually decreases to 1 for old historical data. We conduct experiments on ETTh1, ETTh2, and ILI datasets, which have relatively higher distribution shifts compared to other datasets. Figure 6 shows the test loss of the models in every part of the dataset. As expected, when a distribution shift happens, the test loss increases. However, FrAug can mitigate such performance degradation of the model under distribution shift. Notably, in Figure 6(c), when the distribution shift happens in the 8th part (refer to Figure 7 for visualization). With FrAug, the test loss decreases in the 9th part, while the test loss of the baseline solution keeps increasing without FrAug. This indicates that FrAug helps the model fit into the new distribution quickly. We present more quantitative results in the Appendix.