Compressing Neural Networks with the Hashing Trick

Compressing Neural Networks with the Hashing Trick

19 Apr 2015 | Wenlin Chen*, James T. Wilson*, Stephen Tyree*,† Kilian Q. Weinberger*, Yixin Chen*
The paper "Compressing Neural Networks with the Hashing Trick" by Wenlin Chen introduces HashedNets, a novel network architecture designed to reduce the memory footprint of deep neural networks. The authors address the challenge of deploying deep learning models on mobile devices, which have limited memory capacity. HashedNets leverages the inherent redundancy in neural network weights by using a hash function to group connection weights into buckets, allowing all connections within the same bucket to share a single parameter value. This approach significantly reduces the number of parameters while preserving generalization performance. The hashing procedure does not introduce additional memory overhead and can be integrated with standard backpropagation during training. The paper demonstrates that HashedNets can reduce the storage requirements of neural networks by a substantial amount, making them more suitable for deployment on mobile devices. Experimental results on various benchmark datasets show that HashedNets maintain or improve prediction accuracy compared to other compression methods, such as low-rank decomposition and random edge removal. The authors also explore the benefits of "inflating" the network architecture by reusing parameter values multiple times, achieving better performance with an inflation factor of 8-16 times.The paper "Compressing Neural Networks with the Hashing Trick" by Wenlin Chen introduces HashedNets, a novel network architecture designed to reduce the memory footprint of deep neural networks. The authors address the challenge of deploying deep learning models on mobile devices, which have limited memory capacity. HashedNets leverages the inherent redundancy in neural network weights by using a hash function to group connection weights into buckets, allowing all connections within the same bucket to share a single parameter value. This approach significantly reduces the number of parameters while preserving generalization performance. The hashing procedure does not introduce additional memory overhead and can be integrated with standard backpropagation during training. The paper demonstrates that HashedNets can reduce the storage requirements of neural networks by a substantial amount, making them more suitable for deployment on mobile devices. Experimental results on various benchmark datasets show that HashedNets maintain or improve prediction accuracy compared to other compression methods, such as low-rank decomposition and random edge removal. The authors also explore the benefits of "inflating" the network architecture by reusing parameter values multiple times, achieving better performance with an inflation factor of 8-16 times.
Reach us at info@study.space
Understanding Compressing Neural Networks with the Hashing Trick