12 Mar 2024 | Bohan Liu, Zijie Zhang, Peixiong He, Zhensen Wang, Yang Xiao, Ruimeng Ye, Yang Zhou, Wei-Shinn Ku, Bo Hui
The Lottery Ticket Hypothesis (LTH) posits that a dense neural network contains a highly sparse subnetwork, or "winning ticket," that can achieve better performance when trained in isolation. This hypothesis has been both empirically and theoretically validated, but challenges remain in efficiency, scalability, and the lack of standardized experimental settings. This survey provides an in-depth review of LTH research, categorizing existing works into eight topics: theory, special models, experimental insights, algorithms, efficiency, relations with other topics, open issues, and applications. It discusses the theoretical foundations, extensions to specific models like GNNs and Transformers, and experimental insights into pruning techniques. The survey also highlights efficient strategies to reduce computational costs and explores the transferability of winning tickets across different tasks and domains. Finally, it concludes with open issues and potential research directions, aiming to serve as a valuable resource for researchers and practitioners in the field of pruning technologies.The Lottery Ticket Hypothesis (LTH) posits that a dense neural network contains a highly sparse subnetwork, or "winning ticket," that can achieve better performance when trained in isolation. This hypothesis has been both empirically and theoretically validated, but challenges remain in efficiency, scalability, and the lack of standardized experimental settings. This survey provides an in-depth review of LTH research, categorizing existing works into eight topics: theory, special models, experimental insights, algorithms, efficiency, relations with other topics, open issues, and applications. It discusses the theoretical foundations, extensions to specific models like GNNs and Transformers, and experimental insights into pruning techniques. The survey also highlights efficient strategies to reduce computational costs and explores the transferability of winning tickets across different tasks and domains. Finally, it concludes with open issues and potential research directions, aiming to serve as a valuable resource for researchers and practitioners in the field of pruning technologies.