Kolmogorov-Arnold Networks (KANs)
for Time Series Analysis

Contrary to MLPs, which are based on universal approximation theorem, KANs rely on the Kolmogorov-Arnold representation theorem, also known as the Kolmogorov-Arnold superposition theorem. A fundamental result in the theory of dynamical systems and ergodic theory. It was independently formulated by Andrey Kolmogorov and Vladimir Arnold in the mid-20th century.

The theorem states that any multivariate continuous function $f$, which depends on $\mathbf{x}=[x_{1},x_{2},…,x_{n}]$, on a bounded domain, can be represented as the finite composition of simpler continuous functions, involving only one variable. Formally, a real, smooth, and continuous multivariate function $f(\mathbf{x}):[0,1]^{n}\rightarrow\mathbb{R}$ can be represented by the finite superposition of univariate functions [17]:

where $\Phi_{i}:\mathbb{R}\to\mathbb{R}$ and $\phi_{i,j}:[0,1]\to\mathbb{R}$ denote the so-called outer and inner functions, respectively. One might initially perceive this development as highly advantageous for ML. The task of learning a high-dimensional function simplifies to learning a polynomial number of one dimensional functions. Nevertheless, these 1-dimensional functions can exhibit non-smooth characteristics, rendering them potentially unlearnable in practical contexts. As a result of this problematic behavior, the Kolmogorov-Arnold representation theorem has been traditionally disregarded in machine learning circles, recognized as theoretically solid, but ineffective in practice. Unexpectedly, the theoretical result in [16] has recently emerged as a potential game changer, paving the way for new network architectures, inspired by the Kolmogorov-Arnold theorem.

The authors in [16] mention that equation (3) has two layers of non-linearities, with $2n+1$ terms in the middle layer. Thus, we only need to find the proper functions inner univariate functions $\phi_{i,j}$ and $\Phi_{i}$ that approximate the function. The one dimensional inner functions $\phi_{i,j}$ can be approximated using B-splines. A spline is a smooth curve defined by a set of control points or knots. Splines are often used to interpolate or approximate data points in a smooth and continuous manner. A spline is defined by the order $k$ ($k=3$ is a common value), which refers to the degree of the polynomial functions used to interpolate or approximate the curve between control points. The number of intervals, denoted by $G$, refers to the number of segments or subintervals between adjacent control points. In spline interpolation, the data points are connected by these segments to form a smooth curve (of $G+1$ grid points). Although splines other than B-splines could also be considered, this is the approach proposed in [16]. Equation (3) can be represented as a 2-layer (or analogous 2-depth) network, with activation functions placed at the edges (instead of at the nodes) and nodes performing a simple summation. Such two-layer network is too simplistic to effectively approximate any arbitrary function with smooth splines. For this reason, reference [16] extends the ideas discussed above by proposing a generalized architecture with wider and deeper KANs.

A KAN layer is defined by a matrix $\mathbf{\Phi}$ [16] composed by univariate functions $\{\phi_{i,j}(\cdot)\}$ with $i=1,...,N_{in}$ and $j=1,...,N_{out}$, where $N_{in}$ and $N_{out}$ denote the number of inputs and the number of outputs, respectively, and $\phi_{i,j}$ are the trainable spline functions described above. Note according to the previous definition, the Kolmogorov-Arnold representation theorem presented in Section II-A can be expressed as a two-layer KAN. The inner functions constitute a KAN layer with $N_{in}=n$ and $N_{out}=2n+1$, while the external functions constitute another KAN layer with $N_{in}=2n+1$ and $N_{out}=1$.

Let us define the shape of a KAN by $[n_{1},...,n_{L+1}]$, where $L$ denotes the number of layers of the KAN. It is worth noting the Kolmogorov-Arnold theorem is defined by a KAN of shape $[n,2n+1,1]$. A generic deeper KAN can be expressed by the composition $L$ layers:

Notice that all the operations are differentiable. Consequently, KANs can be trained with backpropagation. Despite their elegant mathematical foundation, KANs are simply combinations of splines and MLPs, which effectively exploit each other’s strengths while mitigating their respective weaknesses. Splines stand out for their accuracy on low-dimensional functions and allow transition between various resolutions. Nevertheless, they suffer from a major dimensionality problem due to their inability to effectively exploit compositional structures. In contrast, MLPs experience a lower dimensionality problem, due to their ability to learn features, but exhibit lower accuracy than splines in low dimensions due to their inability to optimize univariate functions effectively. KANs have by their construction 2 levels of degrees of freedom. Consequently, KANs possess the capability not only to acquire features, owing to their external resemblance to MLPs, but also to optimize these acquired features with a high degree of accuracy, facilitated by their internal resemblance to splines. To learn features accurately, KANs can capture compositional structure (external degrees of freedom), but also effectively approximate univariate functions (internal degrees of freedom with the splines). It should be noted that by increasing the number of layers $L$ or the dimension of the grid $G$, we are increasing the number of parameters and, consequently, the complexity of the network. This approach constitutes an alternative to traditional DL models, which are currently relying on MLP architectures and motivates our extension of this work.

We frame our traffic forecasting problem as a supervised learning framework consisting of a training dataset with input-output $\{\mathbf{x}_{t_{0}-c:t_{0}-1},\mathbf{y}_{t_{0}:T}\}$ in the condition and prediction lengths. We want to find $f$ that approximates $\mathbf{y}_{t_{0}:T}$, i.e., $\mathbf{y}_{t_{0}:T}\approx f(\mathbf{x}_{t_{0}-c:t_{0}-1})$. For ease of notation, we describe our framework as a two-layer (2-depth) KAN [$N_{i}$, $n$, $N_{o}$](note that to comply with the original paper notation, the input layer is not accounted as a layer per se). The output and input layers will be comprised of $N_{o}$, and $N_{i}$ nodes corresponding to the total amount of time steps in (1) and (2), while the transformation/hidden layer of $n$ nodes. The inner functions constitute a KAN layer with $N_{in}=N_{i}$ and $N_{out}=n$, while the external functions constitute another KAN layer with $N_{in}=n$ and $N_{out}=N_{o}$. Our KAN can be expressed by the composition of 2 layers:

where the output functions $\Phi_{2}$ generates the $N_{o}$ outputs values corresponding to (1) by doing the transformation from the previous layers, i.e, we predict the $T$ time steps. The proposed network can be used to forecast future traffic data in the prediction length, based solely on the context length.

Fig. 2 shows a generic representation for any arbitrary number of layers $L$ as presented in (4).

We analyze the forecasting performance in the prediction length for different beams over the test set. Figures 3a-c depicts the real traffic value used as input (in green) to the networks, the expected output prediction length (in blue) and the values predicted values using a KAN (in red) and MLP (in purple) of depth 4 both – see Table I for details on model configuration. In general, our results show that the predictions obtained using KANs better approximates the real traffic values than the predictions obtained using traditional MLPs.

This is particularly evident in Figure 3(a). Here, KAN accurately matches rapid changes in traffic volume, which the MLP models sometimes moderately over/under-predicted, as the last part of the forecast shows. This capability suggests that KANs are better suited to adapt to sudden shifts in traffic conditions, a critical aspect of effective traffic management.

Additionally, the responsiveness of KANs is particularly noticeable in Figure 3(b) during fast changing traffic conditions. KAN shows a rapid adjustment to its forecast that is closely aligned with the actual traffic pattern. This is particularly noticeable in the last $6$ hours of the prediction length where MLP exhibits a lag failing to capture these immediate fluctuations, which shows its worse performance to capture dynamic traffic variations. Further analysis is shown in Figure 3(c), where traffic conditions are more variable and intense, demonstrated the robustness of KAN in maintaining high performance despite the complexity and higher volume. This robustness suggests that KANs can manage different scales and intensities of traffic data more effectively than MLPs, making them more reliable for deployment in varied traffic scenarios.

To further quantify the performance and advantages of using KANs for the satellite traffic forecasting task we show Table LABEL:tab:model_outputs. It shows a detailed comparison of MLPs and KANs different architectures used for evaluation over all the beams. The table displays the Mean Squared Error (MSE), Root Mean Squared Error (RMSE), Mean Absolute Error (MAE), Mean Absolute Percentage Error (MAPE), and the number of trainable parameters for each model. Analyzing the error metrics, it becomes clear that KANs outperform MLPs, where the KAN (4-depth) is the best in performance. Its lower values in MSE and RMSE indicates its better ability to predict traffic volumes with lower deviation. Similarly, its lower values in MAE and MAPE suggests that KANs not only provides more accurate predictions but also maintains consistency across different traffic volumes, which is crucial for practical traffic forecasting scenarios.

Furthermore, the parameter count reveals a significant difference in model complexity. KAN models are notably more parameter-efficient, with KAN (4-depth) utilizing only 109k parameters compared to 329k parameters for MLP (4-depth) or 238k for MLP (3-depth). This reduced complexity suggests that KANs can achieve higher or comparable forecasting accuracy with simpler and potentially lighter models. Such efficiency is especially valuable in scenarios where computational resources are limited or where rapid model deployment is required. The results also show that with an augmentation of 16k parameters in KAN, the performance can be significantly improved, contrary to MLPs which an increment of 91k parameters does not showcase a significant improvement.

From a technical perspective, KANs leverage a theoretical foundation that provides an intrinsic advantage in modeling complex, non-linear patterns typical in traffic systems. This capability likely contributes to their flexibility and accuracy in traffic forecasting. The consistency in performance across diverse conditions also suggests that KANs have strong generalization capabilities, which is essential for models used in geographically varied locations under different traffic conditions. Moreover, besides obtaining lower error rates, our results also suggest that KANs can do so with considerably smaller number of parameters than traditional MLP networks.

We provide an insightful analysis of how different configurations of nodes and grid sizes affect the performance of KANs, particularly in the context of traffic forecasting. For this analysis, we designed 3 KANs (2-depth) $[168,n,24]$ with $n\in\{5,10,20\}$ and varying grids $G\in\{5,10,20\}$ for a $k=3$ order B-spline. These results are shown during training time.

Figure 4 shows a clear trend where increasing the number of nodes generally results in lower loss values. This indicates that higher node counts are more effective at capturing the complex patterns in traffic data, thus improving the performance. For instance, configurations with $n=20$ demonstrate significantly lower losses across all grid sizes compared to those with fewer nodes.

Similarly, the grid size within the splines of KANs has a notable impact on model performance. Larger grid sizes, when used with a significant amount of nodes ($n\in\{10,20\}$), consistently result in better performance. However, when the amount of nodes is low ($n=5$) the extra complexity of the grid size shows the opposite effect. When having a significant amount of nodes larger grids likely provide a more detailed basis for the spline functions, allowing the model to accommodate better variations in the data, which is crucial for capturing complex temporal traffic patterns.

The best performance is observed in configurations that combine a high node count with a large grid size, such as the $n=20$, and $G=20$ setup. This combination likely offers the highest degree of flexibility and learning capacity, making it particularly effective for modeling the intricate dependencies found in traffic data. However, this superior performance comes at the cost of potentially higher computational demands and longer training times, as more trainable parameters are included.

These findings imply that while increasing nodes and grid sizes can significantly enhance the performance of KANs, these benefits must be weighed against the increased computational requirements. For practical applications, particularly in real-time traffic management where timely responses are critical, it is essential to strike a balance. An effective approach could involve starting with moderate settings and gradually adjusting the nodes and grid sizes based on performance assessments and computational constraints. Besides, we want to highlight that for this study continual learning was not assessed, a possibility mentioned in the original paper [16].