This paper explores the use of linear support vector machines (SVMs) as an alternative to the softmax activation function in deep learning models. The authors demonstrate that replacing the softmax layer with a linear SVM can lead to significant improvements in performance on popular datasets such as MNIST, CIFAR-10, and the ICML 2013 face expression recognition challenge. The key advantage of using L2-SVMs is their margin-based loss function, which provides better regularization effects compared to the cross-entropy loss used in softmax models. The paper also discusses the theoretical and practical aspects of combining SVMs with deep neural networks, including the optimization of the primal problem and the backpropagation of gradients. Experimental results show that the DLSVM (deep learning using L2-SVMs) model outperforms the standard softmax model in terms of accuracy and generalization. The authors conclude that the switch from softmax to SVMs is a simple and effective approach for classification tasks in deep learning.This paper explores the use of linear support vector machines (SVMs) as an alternative to the softmax activation function in deep learning models. The authors demonstrate that replacing the softmax layer with a linear SVM can lead to significant improvements in performance on popular datasets such as MNIST, CIFAR-10, and the ICML 2013 face expression recognition challenge. The key advantage of using L2-SVMs is their margin-based loss function, which provides better regularization effects compared to the cross-entropy loss used in softmax models. The paper also discusses the theoretical and practical aspects of combining SVMs with deep neural networks, including the optimization of the primal problem and the backpropagation of gradients. Experimental results show that the DLSVM (deep learning using L2-SVMs) model outperforms the standard softmax model in terms of accuracy and generalization. The authors conclude that the switch from softmax to SVMs is a simple and effective approach for classification tasks in deep learning.