THE LOTTERY TICKET HYPOTHESIS: FINDING SPARSE, TRAINABLE NEURAL NETWORKS

THE LOTTERY TICKET HYPOTHESIS: FINDING SPARSE, TRAINABLE NEURAL NETWORKS

4 Mar 2019 | Jonathan Frankle, Michael Carbin
The paper introduces the *lottery ticket hypothesis*, which posits that dense, randomly initialized neural networks contain subnetworks (*winning tickets*) that can be trained effectively in isolation. These winning tickets have initial weights that facilitate learning, and when reinitialized, they perform poorly. The authors present an algorithm to identify these winning tickets and demonstrate that they can learn faster and achieve higher test accuracy than the original network, even at a significantly smaller size. The hypothesis is supported by experiments on fully-connected and convolutional feed-forward networks, showing that winning tickets are 10-20% of the size of the original networks while maintaining or improving performance. The paper also discusses the implications of the hypothesis for improving training performance, designing better networks, and enhancing theoretical understanding of neural networks.The paper introduces the *lottery ticket hypothesis*, which posits that dense, randomly initialized neural networks contain subnetworks (*winning tickets*) that can be trained effectively in isolation. These winning tickets have initial weights that facilitate learning, and when reinitialized, they perform poorly. The authors present an algorithm to identify these winning tickets and demonstrate that they can learn faster and achieve higher test accuracy than the original network, even at a significantly smaller size. The hypothesis is supported by experiments on fully-connected and convolutional feed-forward networks, showing that winning tickets are 10-20% of the size of the original networks while maintaining or improving performance. The paper also discusses the implications of the hypothesis for improving training performance, designing better networks, and enhancing theoretical understanding of neural networks.
Reach us at info@study.space