Kolmogorov-Arnold Networks (KAN) for Time Series Classification and Robust Analysis

KAN is inspired by Kolmogorov-Arnold representation theory (KAT). It states that any multivariate continuous function defined in a bounded domain can be represented as a finite composition of continuous functions of a single variable and the binary operation of addition. Specifically, if $f$ is a continuous function on a bounded domain $D\subset\mathbb{R}^{n}$, then there exist continuous functions $\phi_{ij}$ and $\psi_{i}$ such that:

where $\phi_{ij}:[0,1]\rightarrow\mathbb{R}$ and $\psi_{i}:\mathbb{R}\rightarrow\mathbb{R}$. It transforms the task of learning a multivariable function into learning a finite number of univariable functions. Compared to MLP, it explicitly provides the number of one-dimensional functions needed for fitting. However, these univariable functions could be non-smooth or even fractal, making it theoretically feasible but practically useless. Nevertheless, Liu et al.[14] found that, by analogy to MLP in neural networks, KAN need not be limited to two layers and finite width to fit all non-linearities. Furthermore, most natural functions tend to be smooth and have sparse structures. These insights suggest that an scalable KAN could become a strong competitor to MLP.

Adversarial attacks involve applying carefully crafted small perturbations $r\in\mathbb{R}^{d}$ to input data $x\in\mathbb{R}^{d}$, leading to significant changes in a model’s output, such as fooling a classifier $f:x\rightarrow\mathbb{R}^{m}$ with the goal of altering the predicted label.

here, the perturbation $r$ is small in magnitude relative to $x$ as indicated by their norms. We normally apply these perturbations to test whether the model can be fooled, thus assessing its robustness. To consider the worst-case scenario, we typically implement the gradient attacks which require knowledge of all the information about the model and the data. Among them, the most widely used method is the Projection Gradient Descent (PGD)[15], the gradient-based iterative attack method, which is the most effective method to evaluate the robustness of models against gradient attacks. This can be characterized by:

here, $t$ is the iteration index, $Clip_{x,\epsilon}\{\cdot\}$ ensures that $x_{adv}^{(t+1)}$ remains within $\epsilon$ of the original input $x$. This method iteratively adjusts $x_{adv}$ to maximize the loss function in the direction that moves it away from its original prediction $y$, while ensuring $x_{adv}$ stays within a small perturbation distance $\epsilon$ from $x$.

A function $f:\mathbb{R}^{m}\to\mathbb{R}^{n}$ is defined to be $\ell_{f}$-locally Lipschitz continuous at radius $r$ if for each $i=1,\ldots,n$, and $\forall\ \|x_{1}-x_{2}\|\leq r$, the following holds:

where $\ell_{f}$ is the local Lipschitz constant. Hereafter, we will refer to it simply as the Lipschitz constant. The Lipschitz constant is directly linked to perturbation stability, which in turn relates to robustness[19].

Assume there is a data distribution $D\subseteq\mathbb{R}^{d}\times\mathbb{R}^{m}$. Our objective is to learn a function $f:x\in\mathbb{R}^{d}\rightarrow y\in\mathbb{R}^{m}$ such that the following risk is minimized as $\hat{R}(f)=\frac{1}{n}\sum_{i=1}^{n}\|y_{i}-f(x_{i})\|$. The purpose of the KAN is to learn such a representation of $f$, thereby minimizing the objective loss. The original KAN used a two-layer structure, while Liu, et al. [14] extended to arbitrary width and depth. In contrast to MLP, the activation function are placed on edges instead of the neurons, KAN use $3^{rd}$-order B-spline ($k=3$) functions for fitting, which allows learning sophisticated activation function by controlling the weight of each basis. In this case, the neuron $q$ in the layer $l+1$ can be represented as :

where $x_{l,p}$ is the input from an arbitrary neuron $p$ in the previous layer $l$. The input from all $n$ neurons in the previous layer $l$ undergoes a nonlinear transformation produced by a learnable B-spline combination, where $G$ is the grid size which determines the number of B-spline bases ($k+G$). This is followed by a weighted summation to obtain the $q^{th}$ output of $x_{l+1,q}^{spline}$. Additionally, KAN introduce a base function similar to residual connections, using weighted $silu$, to stabilize optimization, which can be represented as:

Therefore, the output of the $q^{th}$ neuron in layer $l+1$ can be represented as:

For a multi-layer KAN, the final output can be represented as a nested structure of layers:

where $\Psi_{l}$ denotes the $l^{th}$ layer, which includes the combination of the above two operations: a $spline$ transformation and a base activation $silu$. Fig. 1 illustrates a three-layer KAN structure with the architecture [3-5-3-1], clearly depicting how KAN operate.

We constructed classifiers using KAN, similar to the structure shown in fig. 1. Due to the setting of the B-spline fitting interval being $[-1,1]$, the data distribution outside this interval will not achieve an effective fitting. Instead of directly adopting the method proposed by Liu et al., which suggested updating the grid interval according to data distribution, We employed a more straightforward approach. We fixed the B-Spline grid interval to $[-1,1]$ throughout the process, and applied batch normalization to keep the distribution within $[-1,1]$ in each KAN Layer, to ensure the data distribution conforms to the grid and optimize the training process. Thus, to build a KAN for TSC, we adopted a 3-layer structure with the output transformed to the number of classes, having an architecture of [d-d-128-m], where $d$ is the sequence length and $m$ is the number of classes. Meanwhile, We compared KAN with MLP that had the same number of parameters and neurons per layer, as well as networks where the last layer of KAN was replaced with MLP (KAN_MLP) and the last layer of MLP was replaced with KAN (MLP_KAN). The experimental design is shown in tab. 1.

Fig. 2 (a) and (b) show the accuracy and F1 distribution across 128 UCR datasets for the five models respectively. We observe that these five models achieve relatively similar performance across the 128 datasets, both in terms of F1 score and accuracy. However, KAN performs slightly better overall. This conclusion is also supported by the results shown from the critical diagrams in Fig. 3, where only KAN and MLP_L in the same parameters exhibit statistically significant differences. In the critical diagram, KAN ranks the highest, indicating its strong fitting capability and demonstrating that it can achieve performance comparable to, or even better than, traditional neural networks on the benchmark time series datasets.

KAN have a more complex structure compared to MLP, due to the combinations of base and spline functions. The different grid sizes of spline functions have varying impacts on performance. To evaluate their influences, we investigated three configurations of KAN as follows:

1. 1.

   Complete KAN with different grid sizes: 1, 5, 50

2. 2.

   KAN with only the wpline component, with different grid sizes: 1, 5, 50

3. 3.

   KAN with only the base component

Tab. 2 presents the overall performance of these three configurations across 128 datasets. We observed that an excessively large grid size leads to performance degradation, regardless of whether it is in the complete KAN or without the base function. In the complete KAN, there is little difference in performance with smaller grid sizes. For grid size = 1, nearly 50% of the datasets achieved over 80% accuracy, whereas for grid size = 50, this value drops to less than 70%. In the KAN without the base function, overall performance significantly declines as the grid size increases. Particularly, for grid size = 50, the accuracy of 50% of the datasets is below 43.15%. Additionally, we found that the performance of KAN without the spline function is close to that of the KAN network with only the spline function and with grid size = 1. This indicates that the fitting capability of KAN largely comes from the simple activation functions, suggesting that complex B-spline combinations may lead to optimization difficulties. To explain the result above, we analyzed the CBF dataset, which exhibited results similar to the overall trend as shown in tab. 3. Most results are comparable, except for the model with only the Spline function and a grid size of 50, which performed significantly worse. We analyzed the output results of the two parts at the last layer as shown in the fig. 4.

We observed two phenomena across both the training and testing sets: First, the output values of the spline are relatively smaller and more concentrated compared to those of the base configuration. This indicates that the spline’s contribution to the final decision is less significant than that of the base, suggesting that the base configuration plays a more critical role in decision-making. Second, the most significant difference between fig. 4(c) and 4(d) is the distinct distribution observed when the base component is removed. When the grid size is set to 1, the output distributions for these three configurations are similar, exhibiting two prominent peaks on both the positive and negative sides, with the negative peak being higher and more numerous. This pattern occurs because the CBF dataset has 3 categories, thus, when one class is predicted with high confidence, the other two classes tend to output negative values. However, this scenario changes drastically at a grid size of 50. Here, the spline output shows only a single peak concentrated around zero both in the training and testing set, corresponding to the lower accuracy observed in tab. 2 for a grid size of 50 (without Base). This further confirms that an excessively large grid size complicates the network’s optimization.

We also found that KAN demonstrate better robustness compared to MLP. We performed PGD untargeted attacks on the aforementioned five models, with $\epsilon$ ranging from 0.05 to 0.5. The results consistently show that KAN significantly outperform MLP. Fig. 5 illustrates the ASR of PGD on these models. We observe that KAN and MLP_KAN exhibit remarkable robustness compared to the other three models, with this advantage increasing as $\epsilon$ grows. Specifically, at $\epsilon=0.5$, the MLP_KAN model shows the best robustness among the five, with the ASR on 50% of the dataset remaining below approximately 75%, whereas the ASR for MLP, MLP_L, and KAN_MLP approach 1 for nearly 50% of the dataset. From fig. 6, it is evident that KAN and MLP_KAN demonstrate significantly different robustness across 128 datasets compared to the other three models.

To explain this phenomenon, we obtained the Lipschitz constants for the KAN, MLP, and MLP_KAN models. Fig. 7 shows the distribution of Lipschitz constant differences across 128 datasets. Fig. 8(a) and 8(b) both indicate that the model with KAN layer generally results in a decrease in the Lipschitz constants for most datasets, which is consistent with the observations in fig. 5. The combination of spline functions produced by KAN with a low grid size tends to be smooth and flat, making it difficult for small changes in input to cause significant changes in the output $y$. Additionally, as shown in the previous fig. 4, the output distribution of the B-spline function is more narrow compared to the combination of activation and linear functions, thus contributing minimally to the output. This could be the primary reason why KAN is more robust under attack.

However, we observed the opposite result in another experiment. Fig. 8 shows that the Lipschitz constant corresponding to a larger grid size is greater. This is evident because as the number of grids increases, the slopes of the spline basis also increase, making the overall B-spline function more likely to produce larger slopes. However, the results indicate that a larger Lipschitz constant leads to greater robustness, with 104 out of 128 datasets having an ASR less than or equal to that corresponding to a grid size of 1, demonstrating stronger robustness. We preliminarily believe this might be because the value distribution produced by the B-spline is much smaller compared to the base function, thus the base contributes the majority in decision-making. However, the gradients of the B-spline with a larger grid size are substantial, thus during PGD attacks, the gradient is mainly provided by the B-spline part, which might not necessarily provide the correct direction compared to the base. Therefore, networks with larger Lipschitz constants exhibit stronger robustness. This may also further imply why models are difficult to optimize as the grid size increases.