Binarized Neural Networks: Training Neural Networks with Weights and Activations Constrained to +1 or -1

Binarized Neural Networks: Training Neural Networks with Weights and Activations Constrained to +1 or -1

17 Mar 2016 | Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, Yoshua Bengio
This paper introduces Binarized Neural Networks (BNNs), a type of neural network where weights and activations are constrained to +1 or -1 during runtime. The authors propose a method to train BNNs, which uses binary weights and activations during the forward pass to reduce memory usage and computational complexity. They demonstrate that BNNs can achieve nearly state-of-the-art results on the MNIST, CIFAR-10, and SVHN datasets using both Torch7 and Theano frameworks. The binary nature of BNNs allows for efficient computation using bitwise operations, which can significantly improve power efficiency. The authors also develop a binary matrix multiplication GPU kernel that enables their MNIST BNN to run 7 times faster than an unoptimized GPU kernel without any loss in classification accuracy. The paper also discusses the use of shift-based batch normalization and AdaMax optimization to further enhance the performance of BNNs. The results show that BNNs are not only efficient in terms of power consumption but also maintain high accuracy, making them a promising approach for deploying deep neural networks on low-power devices.This paper introduces Binarized Neural Networks (BNNs), a type of neural network where weights and activations are constrained to +1 or -1 during runtime. The authors propose a method to train BNNs, which uses binary weights and activations during the forward pass to reduce memory usage and computational complexity. They demonstrate that BNNs can achieve nearly state-of-the-art results on the MNIST, CIFAR-10, and SVHN datasets using both Torch7 and Theano frameworks. The binary nature of BNNs allows for efficient computation using bitwise operations, which can significantly improve power efficiency. The authors also develop a binary matrix multiplication GPU kernel that enables their MNIST BNN to run 7 times faster than an unoptimized GPU kernel without any loss in classification accuracy. The paper also discusses the use of shift-based batch normalization and AdaMax optimization to further enhance the performance of BNNs. The results show that BNNs are not only efficient in terms of power consumption but also maintain high accuracy, making them a promising approach for deploying deep neural networks on low-power devices.
Reach us at info@study.space
[slides and audio] Binarized Neural Networks