Exploring Kolmogorov-Arnold networks for realistic image sharpness assessment

KAT-inspired KAN treats a multivariate function as learnable uni-variate spline functions on edges [18]. A KAN layer can be defined by a matrix of uni-variate functions $\Phi=\{\phi_{q,p}\}$ in which $p=1,2,\ldots,n_{\text{in}}$ and $q=1,2,\ldots,n_{\text{out}}$. Here, $n_{\text{in}}$ is the input dimension, $n_{\text{out}}$ is the output dimension, and $\phi_{q,p}$ is a learnable function. The activation of each node in layer $l+1$ can be computed as

And thus, the $L$-layered KAN can be generally described as

and implemented in a layer-to-layer connection form [19].

The flexibility of KAN structure allows for diverse implementations using different activation functions on edges. The TaylorKAN and other involved KANs are described.

Four databases (BID2011 [26], CID2013 [27], CLIVE [28] and KonIQ-10k [29]) with realistic distortions are analyzed. Table I shows the number (#) of distorted images, the year of data availability, and the score ranges of the databases.

Two sets of features are prepared. One contains 15 mid-level features per image that are the output of BISA indicators [8]. The other set includes 2048 deeply learned features which are derived from the last full-connection layer of the pre-trained ResNet50 [30]. The procedure is similar as [17].

Besides KANs, SVR and MLP are tested. Assume features ($X=\{\vec{x_{i}}\}_{i=1}^{l}$) and scores ($Y=\{y_{i}\}_{i=1}^{l}$) of $l$ samples in the training set, a weighting vector $\boldsymbol{w}$, and a new input vector $\vec{x}$. SVR ($svr$) aims to find a function $R_{svr}(X)$ that maximizes the deviation of $\epsilon$ from the subjective score $y_{i}$ of training samples. In Eq. 13, $\zeta(X)$ is a nonlinear function, and $\gamma$ is a bias.

MLP ($mlp$) is designed with different numbers of the hidden layers in terms of different feature inputs. Its parameters are optimized by minimizing the difference between the output of model $R_{mlp}$ and the ground truth $Y$.

The performance is evaluated by using Pearson linear correlation coefficient (PLCC) and Spearman rank order correlation coefficient (SRCC). Higher values indicate better performance.

Specifically, PLCC is computed after a five-parameter nonlinear mapping between objective and subjective scores (Eq. 15) in which $s$ denotes the predicted score, $f(s)$ is the mapped score, and $\{q_{i}\}_{i=1}^{5}$ are the fitting parameters. PLCC values are calculated between subjective scores and mapped scores.

Training time is used to evaluate the efficiency of involved score prediction models. The training time for SVR is recorded in seconds ($s$), and the other models are measured in iterations per second ($i/s$), where a lower value indicates poorer computational efficiency.

In each experiment, a database is randomly split into three subsets for training (70% samples), validation (15% samples) and testing (the remaining samples) of BIQA indicators.

For fair comparison, the KANs and MLP are configured with 3 hidden layers ([15, 26, 18, 12, 1]) for mid-level inputs and 4 hidden layers ([2048, 1536, 1024, 256, 128, 1]) for high-level feature inputs. TaylorKAN is implemented with quadratic approximation, and the other KAN models use default settings. To SVR, the radial basis function (RBF) kernel is used, and the other parameters are set with default values.

During training, an early stopping mechanism is used even though 500 iterations are pre-defined. The patience parameter is set to 20, and thus, the training will be terminated early if the validation loss fails to improve for 20 consecutive epochs. Throughout the process, the metrics to the learning rate that yields the highest sum of PLCC and SRCC are reported.

The codes are implemented on an Ubuntu operating system (version 22.04) using Python (version 3.12), PyTorch (version 2.3.0), and CUDA (version 12.1). The algorithms are executed on a GPU (A100-PCIE-40GB) with 72 GB RAM. The project is available at https://github.com/CUC-Chen/KAN4IQA.

Table II shows that compared to SVR, TaylorKAN achieves better prediction on BID2011, CID2013 and KonIQ-10k, and its PLCC value is higher on CLIVE. The other KAN models obtain better performance on BID2011, competitive results on CID2013 and KonIQ-10k, but worse prediction on CLIVE. Among the KANs, ChebyKAN is the fastest, achieving over 30 $i/s$ on BID2011 and CID2013, and TaylorKAN demonstrates slightly lower but comparable computational efficiency.

When using the 15 mid-level features for quality representation, TaylorKAN achieves the best performance, attributed to several factors. Firstly, it surpasses SVR by leveraging the Taylor series for hierarchical approximation, and meanwhile, a multi-layer network is implemented for deeper feature learning. Secondly, it outperforms other involved KANs, possibly because quadratic approximation enables more precise feature fitting. Thirdly, it achieves higher metric values than MLP, since the former uses additional activation functions on edges. At last, the simplicity of the quadratic approximation enables TaylorKAN to perform data fitting at a relatively faster pace.

Setting SVR as the baseline, Table III suggests that KANs achieve higher values on BID2011, worse results on CID2013 and CLIVE, and close performance on KonIQ-10k in general. Notably, BSRBF_KAN and WavKAN are comparable to SVR on CID2013 and CLIVE, while TaylorKAN, BSRBF_KAN, JacobiKAN and HermiteKAN are superior on KonIQ-10k. It is found that BSRBF_KAN, WavKAN and TaylorKAN run at less than 10 $i/s$ on BID2011 and CID2013.

When applying the 2048 high-level features for image quality representation, TaylorKAN obtains slightly higher metric values on BID2011 and KonIQ-10k, while worse results on CID2013 and CLIVE than SVR. The sub-optimal performance indicates that it remains challenging for the KANs to handle high-dimensional features. On the one hand, the KAT (Eq. 1) implies that the representation of a high-dimensional function requires a large number of uni-variate functions. Consequently, the accumulation of errors and over-fitting may cause unstable data-driven approximation. On the other hand, whether these high-level features primarily learned for object recognition can be directly utilized to represent image quality has not yet been determined. Therefore, proper post-processing strategies, such as fine-tuning [9] and rectification [17], become important for smoothly transferring these deeply learned features into the representation space of image quality.

Comparison of Table II and Table III reveals that the 2048 high-level features provide more effective quality representation than the 15 mid-level features for score prediction, except on the CID2013 database. Most KANs, such as BSRBF_KAN, ChebyKAN and JacobiKAN, achieve better performance when high-level features are used, while the utilization of the 2048 features dramatically increases the computational time. On the one hand, it should be acknowledged the mid-level features are less effective than the high-level features due to their limited quantity and capacity for quality representation. Specifically, the mid-level features are primarily designed for scoring image sharpness [8], while the high-level features are hierarchically and deeply learned for general recognition task [30]. On the other hand, the KANs perform generally better on CID2013 when using the mid-level features. For example, in the case of JacobiKAN, using mid-level features leads to PLCC 0.808 and SRCC 0.775, whereas using high-level features yields PLCC 0.596 and SRCC 0.605. This finding indicates that the mid-level features are more suitable for the CID2013 database.

The results of TaylorKANs using mid-level features (^a) and high-level features (^b) and several state-of-the-art (SOTA) works are shown in Table IV. These works [11, 10, 12, 13, 14, 15, 16] develop novel CNNs that take images rather than features as input.

A significant performance gap is found between the SOTA works and TaylorKAN in score prediction. The CNNs achieve PLCC $\geq$ 0.86 and SRCC $\geq$ 0.84, which are much higher than those of the evaluated models (Table II and III). These SOTA works benefit not only from advanced architectures of image convolution and feature pooling but also from sophisticated module designs, such as uncertainty learning [11], spatial pooling [13], self-supervised learning [15], and multitask learning [16]. Thus, one way to improve representation power and score prediction performance is to integrate prior knowledge and advanced modules into KANs [20]. When using TaylorKAN for score prediction, high-level features can increase, but not substantially, metric values on three databases. On the one hand, the perceptual similarity is an emergent property shared across deep visual embedding [31] and thus, it is not surprise that deep features are effective for image quality representation [15]. On the other hand, how to figure out the most informative subsets of deep features is important for KANs to enhance effectiveness and efficiency in score prediction, while avoiding the challenges of high-dimensional feature processing.

This study has several limitations. Firstly, a broader range of informative features, potentially numbering in the hundreds or thousands, could be collected and refined by selecting the most relevant ones. These selected features can then be applied in KANs for further data fitting and feature embedding tailored to specific tasks. Secondly, KANs could be integrated into deep learning architectures by replacing the MLP components. Such integration may enhance KANs’ representation capacity in end-to-end optimization. Thirdly, the performance of KANs could be further assessed on other tasks, such as object recognition, to gain deeper insights into their strengths, limitations, and potential applications in feature representation.