Spiking Convolutional Neural Networks for Text Classification

Spiking Convolutional Neural Networks for Text Classification

27 Jun 2024 | Changze Lv, Jianhan Xu, and Xiaoqing Zheng*
This paper presents a two-step method for training spiking neural networks (SNNs) for text classification, combining a "conversion" step and a "fine-tuning" step. The conversion step involves transforming a traditional neural network into an SNN by replacing max-pooling with average-pooling, using ReLU activation functions instead of sigmoid, and converting word embeddings to positive-valued vectors. The fine-tuning step uses surrogate gradients to update the SNN weights. The proposed method achieves comparable results to deep neural networks (DNNs) on multiple text classification datasets for both English and Chinese, with significantly reduced energy consumption. Additionally, the SNNs trained with this method are more robust to adversarial attacks compared to DNNs. The paper also includes an ablation study to demonstrate the effectiveness of each step and explores the impact of hyperparameters on performance.This paper presents a two-step method for training spiking neural networks (SNNs) for text classification, combining a "conversion" step and a "fine-tuning" step. The conversion step involves transforming a traditional neural network into an SNN by replacing max-pooling with average-pooling, using ReLU activation functions instead of sigmoid, and converting word embeddings to positive-valued vectors. The fine-tuning step uses surrogate gradients to update the SNN weights. The proposed method achieves comparable results to deep neural networks (DNNs) on multiple text classification datasets for both English and Chinese, with significantly reduced energy consumption. Additionally, the SNNs trained with this method are more robust to adversarial attacks compared to DNNs. The paper also includes an ablation study to demonstrate the effectiveness of each step and explores the impact of hyperparameters on performance.
Reach us at info@study.space