This paper proposes a Structured Sparsity Learning (SSL) method to regularize the structures of deep neural networks (DNNs), including filters, channels, filter shapes, and layer depth. SSL learns a compact structure from a larger DNN to reduce computation cost and achieves hardware-friendly structured sparsity for efficient DNN evaluation. Experimental results show that SSL achieves on average 5.1× and 3.1× speedups for convolutional layers of AlexNet on CPU and GPU, respectively, compared to non-structured sparsity. SSL also improves classification accuracy by regularizing DNN structures. For example, it reduces the number of layers in ResNet from 32 to 18 while improving accuracy from 91.25% to 92.60%. For AlexNet, SSL reduces error by about 1%. SSL is a generic regularization method that adaptively adjusts multiple structures in DNNs, including filters, channels, filter shapes, and layer depth. It combines structure regularization for classification accuracy with locality optimization for computation efficiency. SSL can achieve well-regularized models with improved accuracy and significantly accelerated computation. The method is evaluated on three databases: MNIST, CIFAR-10, and ImageNet. On MNIST, SSL reduces the number of filters and channels while maintaining similar accuracy. On CIFAR-10, SSL reduces the size of the weight matrix and achieves good speedups without accuracy loss. On ImageNet, SSL achieves significant speedups on both CPU and GPU, with an average of 5.1× and 3.1× speedups for CPU and GPU, respectively. SSL outperforms non-structured sparsity methods like ℓ₁-norm in terms of speedup and accuracy. The source code is available at https://github.com/wenwei202/caffe/tree/scnn.This paper proposes a Structured Sparsity Learning (SSL) method to regularize the structures of deep neural networks (DNNs), including filters, channels, filter shapes, and layer depth. SSL learns a compact structure from a larger DNN to reduce computation cost and achieves hardware-friendly structured sparsity for efficient DNN evaluation. Experimental results show that SSL achieves on average 5.1× and 3.1× speedups for convolutional layers of AlexNet on CPU and GPU, respectively, compared to non-structured sparsity. SSL also improves classification accuracy by regularizing DNN structures. For example, it reduces the number of layers in ResNet from 32 to 18 while improving accuracy from 91.25% to 92.60%. For AlexNet, SSL reduces error by about 1%. SSL is a generic regularization method that adaptively adjusts multiple structures in DNNs, including filters, channels, filter shapes, and layer depth. It combines structure regularization for classification accuracy with locality optimization for computation efficiency. SSL can achieve well-regularized models with improved accuracy and significantly accelerated computation. The method is evaluated on three databases: MNIST, CIFAR-10, and ImageNet. On MNIST, SSL reduces the number of filters and channels while maintaining similar accuracy. On CIFAR-10, SSL reduces the size of the weight matrix and achieves good speedups without accuracy loss. On ImageNet, SSL achieves significant speedups on both CPU and GPU, with an average of 5.1× and 3.1× speedups for CPU and GPU, respectively. SSL outperforms non-structured sparsity methods like ℓ₁-norm in terms of speedup and accuracy. The source code is available at https://github.com/wenwei202/caffe/tree/scnn.