Kolmogorov-Arnold Convolutions: Design Principles and Empirical Studies

Kolmogorov-Arnold Convolutions: Design Principles and Empirical Studies

July 2, 2024 | Ivan Drokin
This paper explores the application of Kolmogorov-Arnold Networks (KANs) in computer vision, focusing on their adaptation to convolutional layers. KANs leverage the Kolmogorov-Arnold theorem to incorporate splines into their architecture, offering an alternative to traditional Multi-Layer Perceptrons (MLPs). The paper proposes a parameter-efficient design for KAN convolutional layers and a parameter-efficient fine-tuning algorithm for pre-trained KAN models. It also introduces KAN convolutional versions of self-attention and focal modulation layers. Empirical evaluations on datasets such as MNIST, CIFAR10, CIFAR100, Tiny ImageNet, ImageNet1k, and HAM10000 demonstrate that KAN-based models achieve state-of-the-art results in classification tasks. For segmentation tasks, U-Net-like architectures with KAN convolutions are proposed, achieving state-of-the-art results on BUSI, GlaS, and CVC datasets. The paper also investigates regularization techniques for KANs and provides a design guide for constructing successful KAN convolutional models. Key contributions include the Bottleneck Convolutional Kolmogorov-Arnold layer, which reduces memory requirements, and parameter-efficient fine-tuning algorithms for Gram polynomials. The paper also explores the use of KANs in self-attention and focal modulation layers, demonstrating improved performance in classification tasks. The results highlight the effectiveness of Gram polynomials as a basis function for KANs, the advantages of scaling model width over depth, and the potential of DenseNet-like architectures for very deep networks. The paper concludes that KAN-based convolutional models offer a promising direction for optimizing neural network architectures in computer vision.This paper explores the application of Kolmogorov-Arnold Networks (KANs) in computer vision, focusing on their adaptation to convolutional layers. KANs leverage the Kolmogorov-Arnold theorem to incorporate splines into their architecture, offering an alternative to traditional Multi-Layer Perceptrons (MLPs). The paper proposes a parameter-efficient design for KAN convolutional layers and a parameter-efficient fine-tuning algorithm for pre-trained KAN models. It also introduces KAN convolutional versions of self-attention and focal modulation layers. Empirical evaluations on datasets such as MNIST, CIFAR10, CIFAR100, Tiny ImageNet, ImageNet1k, and HAM10000 demonstrate that KAN-based models achieve state-of-the-art results in classification tasks. For segmentation tasks, U-Net-like architectures with KAN convolutions are proposed, achieving state-of-the-art results on BUSI, GlaS, and CVC datasets. The paper also investigates regularization techniques for KANs and provides a design guide for constructing successful KAN convolutional models. Key contributions include the Bottleneck Convolutional Kolmogorov-Arnold layer, which reduces memory requirements, and parameter-efficient fine-tuning algorithms for Gram polynomials. The paper also explores the use of KANs in self-attention and focal modulation layers, demonstrating improved performance in classification tasks. The results highlight the effectiveness of Gram polynomials as a basis function for KANs, the advantages of scaling model width over depth, and the potential of DenseNet-like architectures for very deep networks. The paper concludes that KAN-based convolutional models offer a promising direction for optimizing neural network architectures in computer vision.
Reach us at info@study.space