Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures

Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures

12 Jul 2016 | Hengyuan Hu, Rui Peng, Yu-Wing Tai, Chi-Keung Tang
Network Trimming is a data-driven neuron pruning approach aimed at creating efficient deep architectures. The method iteratively prunes neurons with low activation values, identified through analysis of their outputs on a large dataset. The key observation is that many neurons in deep networks have mostly zero activations, making them redundant. By pruning these neurons and retraining the network, the algorithm reduces the number of parameters while maintaining or even improving accuracy. The approach involves training a network, then pruning neurons with high zero activation rates (APoZ), followed by retraining to restore performance. This process is repeated until the desired efficiency is achieved. Experiments on LeNet and VGG-16 show that this method can achieve significant parameter compression without loss in accuracy, sometimes even improving it. The method is more effective than traditional connection pruning techniques, as it directly targets redundant neurons rather than connections. It also avoids overfitting by using validation data to assess neuron importance, and the results show that trimmed networks perform well on both validation and test sets, indicating reduced overfitting. Network Trimming lies between high-level network redesign and low-level weight pruning, offering a practical way to optimize neural networks. It can be applied to any mature architecture, combining with weight pruning to significantly reduce complexity. The method has been tested on various networks, achieving high compression rates with minimal performance loss.Network Trimming is a data-driven neuron pruning approach aimed at creating efficient deep architectures. The method iteratively prunes neurons with low activation values, identified through analysis of their outputs on a large dataset. The key observation is that many neurons in deep networks have mostly zero activations, making them redundant. By pruning these neurons and retraining the network, the algorithm reduces the number of parameters while maintaining or even improving accuracy. The approach involves training a network, then pruning neurons with high zero activation rates (APoZ), followed by retraining to restore performance. This process is repeated until the desired efficiency is achieved. Experiments on LeNet and VGG-16 show that this method can achieve significant parameter compression without loss in accuracy, sometimes even improving it. The method is more effective than traditional connection pruning techniques, as it directly targets redundant neurons rather than connections. It also avoids overfitting by using validation data to assess neuron importance, and the results show that trimmed networks perform well on both validation and test sets, indicating reduced overfitting. Network Trimming lies between high-level network redesign and low-level weight pruning, offering a practical way to optimize neural networks. It can be applied to any mature architecture, combining with weight pruning to significantly reduce complexity. The method has been tested on various networks, achieving high compression rates with minimal performance loss.
Reach us at info@study.space
[slides] Network Trimming%3A A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures | StudySpace