Kernel-U-Net: Multivariate Time Series Forecasting
using Custom Kernels

Transformer [16] was initially introduced in the field of Natural Language Processing (NLP) on language translation tasks. It contains a layer of positional encoding, blocks composed of layers of multiple head attentions, and a linear layer with SoftMax activation. As it demonstrated outstanding performance on NLP tasks, many researchers follow this technique route.

Vision Transformers (ViTs) [17] applied a pure transformer directly to sequences of image patches to classify the full image and outperformed CNN based method on ImageNet[18]. Swin Transformer [19] proposed a hierarchical Transformer whose representation is computed with shifted windows. As a shifted window brings greater efficiency by limiting self-attention computation to non-overlapping local windows, it also allows cross-window connection. This hierarchical architecture has the flexibility to model at various scales and has linear computational complexity concerning image size.

In time series forecasting, the researchers were also attracted by transformer-based models. LogTrans [10] proposed convolutional self-attention by employing causal convolutions to produce queries and keys in the self-attention layer. To improve the computation efficiency, the authors propose also a LogSparse Transformer with only $O(L(logL)^{2})$ space complexity to break the memory bottleneck. Informer [3] has an encoder-decoder architecture that validates the Transformer-like model’s potential value to capture individual long-range dependency between long sequences. The authors propose a ProbSparse self-attention mechanism to replace the canonical self-attention efficiently. It achieves the $O(LlogL)$ time complexity and $O(LlogL)$ memory usage on dependency alignments. Pyraformer[20] simultaneously captures temporal dependencies of different ranges in a compact multi-resolution fashion. Theoretically, by choosing parameters appropriately, it achieves concurrently the maximum path length of $O(1)$ and the time and space complexity of $O(L)$ in forward pass. PatchTST employs ProbSparse and a linear projection patch layer to reduce the computational complexity of the transformer to $O(\frac{L}{S}\log(\frac{L}{S}))$, where $S$ is patch size. However, its flattened head still incurs a computational cost of $O(\frac{L^{2}}{S})$.

Meanwhile, another family of transformer-based models combines the transformer with the traditional method in time series processing. Autoformer [2] introduces an Auto-Correlation mechanism in place of self-attention, which discovers the sub-series similarity based on the series periodicity and aggregates similar sub-series from underlying periods. Frequency Enhanced Decomposed Transformer (FEDformer)[11] captures global properties of time series with seasonal-trend decomposition. The authors proposed Fourier-enhanced blocks and Wavelet-enhanced blocks in the Transformer structure to capture important time series patterns through frequency domain mapping.

U-Net is a neural network architecture designed primarily for medical image segmentation [14]. U-net is composed of an encoder and a decoder. At the encoder phase, a long sequence is gradually reduced by a convolutional layer and a max-pooling layer into a latent vector. At the decoder stage, the latent vector is developed by a transposed convolutional layer for generating an output with the same shape as the input. With the help of skip-connection between the encoder and decoder, U-Net can also capture and merge low-level and high-level information easily.

With such a neat structural design, U-Net has achieved great success in a variety of applications such as medical image segmentation [14], biomedical 3D-Image segmentation [21], time series segmentation [22], image super-resolution [23] and image denoising [24]. Its techniques evolve from basic 2D-U-Net to 3D U-Net [21], 1D-U-Net [25] and Swin-U-Net [26] that only use transformer blocks at each layer of U-Net.

In time series processing, U-time [22] is a U-Net composed of convolutional layers for time series segmentation tasks, and YFormer [15] is the first U-Net based model for time series forecasting task. In particular, YFormer applied transformer blocks on each layer of U-Net and capitalized on multi-resolution feature maps effectively.

In time series processing, the increasing size of data degrades the performance of deep models and, crucially, increases the cost of learning them. For example, recurrent models such as RNN, LSTM, GRU have a linear complexity but suffer the gradient vanishing problem when input length increases. Transformer-based block captures better long dependencies but requires $O(L^{2})$ computations in general. To balance the expressiveness of complex models and computational efficiency, researchers investigated hybrid models that merge different modules into the network and hierarchical architectures.

For example, authors in [27] investigated a tree structure model made of bidirectional RNN layers and concatenation layers for skeleton-based action recognition, authors in [28] stacked RNN and LSTM with Attention mechanism for semantic relation classification of text, authors in [29] applied hierarchical LSTM and GRU for document classification. In time series processing, the authors in [30] combine Deep Belief Network (DBN) and LSTM for sleep signal pattern classification.

To meet the demand of balancing the quality of prediction and efficiency in learning Transformer-based models, researchers proposed hierarchical structure in Swin-Transformer [19], pyramidal structure in Pyraformer [20], U-shape structure in Yformer [15] or patch layer in PatchTST [13].

Kernel-U-Net is a U-shape architecture that exponentially reduces the input length at each level, thereby concurrently decreasing the complexity involved in learning long sequences. Kernel-U-Net separates the procedure of partitioning input time series into patches from kernel manipulation, thereby providing the convenience of executing customized kernels. By replacing linear kernels with transformer or LSTM kernels, the model gains enhanced expressiveness, allowing it to capture more complex patterns and dependencies in the data. Notably, when Transformer modules are utilized in the second or higher-level layers, the computation cost remains linear, ensuring efficiency in processing.

We conducted experiments with our proposed Kernel U-Net on 7 public datasets, including 4 ETT (Electricity Transformer Temperature) datasets²²2https://github.com/zhouhaoyi/ETDataset (ETTh1, ETTh2, ETTm1, ETTm2), Weather³³3https://www.bgc-jena.mpg.de/wetter/, Traffic⁴⁴4http://pems.dot.ca.gov and Electricity⁵⁵5https://archive.ics.uci.edu/ml/datasets/ElectricityLoadDiagrams20112014. These datasets have been benchmarked and publicly available on[2] and the description of the dataset is available in [12].

Here, we followed the experiment setting in [12] and partitioned the data to $[12,4,4]$ months for training, validation, and testing respectively for the ETT dataset. The data is split to $[0.7,0.1,0.2]$ for training, validation, and testing for the Weather, Traffic, and Electricity datasets.

We follow the experiment setting in NLinear [12] and take $L$ step historical data as input then forecast $T$ step value in the future where $L\in\mathcal{L}=\{336,720\}$ and $T\in\mathcal{T}=\{96,192,336,720\}$. We replace the last value normalization by mean value normalization for ETT, Electricity, and Weather datasets, and apply instance normalization[32] for the Traffic dataset. We use Mean Squared Error (MSE) and Mean Absolute Error (MAE) for evaluation as mentioned in [2].

We include recent methods: PatchTST[13], NLinear, DLinear[12], FEDformer [11], Autoformer [2], Informer [3], LogTrans [10] Yformer[15]. We merge the result reported in [13] and [12] for taking their best in a supervised setting and execute Yformer with default parameters in its Github⁶⁶6https://github.com/18kiran12/Yformer-Time-Series-Forecasting.

We use a 4-layer K-U-Net for experiments. The list of multiples are respectively {4, 3, 7} and {6, 6, 5} for look-back windows $L\in\{336,720\}$. The bottom patch length is 4 and its width is 1. We reshape the input into $(B\cdot M,L,1)$ for processing with K-U-Net as we follow the channel-independent setting [12] and then reshape it back. The hidden dimensions are 128 for multivariate tasks. The learning rate is selected in $[0.00005,0.001]$. The training epoch is 50 and the patience of early stopping is 10 in general. We use MAE as the loss function.

We propose 4 kernels for experiments with K-U-Net on 7 datasets. By replacing a linear kernel at different layers with other types of kernels, we search for variants that adapt the dataset. As there are too many variants to name, we note them as ”KUN_$<$kernel$>$_$<$replace_layer$>$ ($<$look-back_window$>$)”. For example, KUN_Linear$\_0000$ $(720)$ means that the model is made of linear kernels at all layers and the look-back window size is 720, KUN_Transf$\_0100$ means that a transformer kernel replaces the linear kernel at the second layer, KUN_Linear$\_0110$ means that multilayer perceptron kernels replace the kernels at the second and third layers. We have composed 16 variants with MLP kernels and 7 variants with Transformer and LSTM kernels.

To enumerate all variants that achieve at least once the best result, we report their performance by the average of the top 5 minimum running MSE (Top5MMSE) values on the validation and test set. The search results are reported with a relative score(RS) to the minimum Top5-MMSE:

, where $U$, $V$ are variant models in the models set $\mathcal{V}$, $T$ is a forecasting horizon in $\mathcal{T}$, $L$ is a look back window in $\mathcal{L}$. Relative score notes the best-performed model with 1 and thus helps to identify the high-potential candidates for further fine-tuning examinations.

We observe in Figure 3 that the best variant for ETTh1 dataset is KUN_Linear$\_0010$. Comparing the relative score of the K-U-Net of replace$\_$layer code ($0010$, $0011$), ($0110$, $0111$) and ($0100$, $0101$), we remark that replacing the highest layer with Transformer and LSTM kernel degrades the performance because of overfitting. Furthermore, We observe in Figure 5 that the best variants for the Weather dataset are KUN_Linear$\_0110$, KUN_LSTM$\_0100$ and KUN_Transf$\_0100$. Comparing the relative score of the K-U-Net of replace$\_$layer code ($0010$, $0110$), ($0011$, $0111$) and ($0001$, $0101$) we remark that replacing the second layer with Transformer and LSTM kernel gains the performance for their expressiveness.

Among candidates in the search phase, we empirically choose 3 variants for fine-tuning experiments with 5 runs. The final result shows that the performance of Kernel U-Net exceeds or meets the state-of-the-art methods in multivariate settings in most cases.

We remark that our model improves the MSE performance around $72.92\%$ compared with Yformer and $2.99\%$ compared to PatchTST and NLinear in the multivariate setting (Table I). It is worth noting that K-U-Net achieves similar results on the Electricity dataset with a variant based on MLP kernels.

We examined the computation efficiency of 6 variants of K-U-Net, PatchTST, and Yformer. We execute the models on the ETTh1 and ETTm1 datasets for 10 epochs and measure the average execution time per epoch and the GPU consumption during the training. The hidden dimension is set to 128 for all models. For fair comparison, PatchTST and Yformer contain 2 layers of Transformer block which equals to KUN_Transf$\_0100$. All experiments are executed on a Tesla V100 GPU in a Google Colab environment. Comparing to the PatchTST, KUN_Linear$\_0000$ saves $85.20\%$ and $83.07\%$ memory (Figure 4) and saves $81.82\%$ and $80.55\%$ computation time (Figure 6) on ETTh1 and ETTm1 datasets, KUN_Transf$\_0100$ saves $55.15\%$ and $50.00\%$ memory and saves $28.47\%$ and $26.88\%$ computation time respectively.