Rewrite the Stars

Rewrite the Stars

29 Mar 2024 | Xu Ma¹, Xiyang Dai², Yue Bai¹, Yizhou Wang¹, Yun Fu¹
This paper investigates the star operation (element-wise multiplication) in neural network design, revealing its ability to map inputs into high-dimensional, non-linear feature spaces without increasing network width. The authors demonstrate that the star operation can generate a feature space with approximately (d/√2)^2 linearly independent dimensions, similar to polynomial kernel functions. By stacking multiple layers, the star operation can exponentially increase the implicit feature dimensions, achieving nearly infinite dimensions within a compact network. This unique property makes the star operation highly efficient and suitable for compact network designs. The authors introduce StarNet, a simple and efficient network that leverages the star operation to achieve strong performance with minimal design complexity. StarNet outperforms several state-of-the-art efficient models in terms of accuracy and speed, achieving 73.5% top-1 accuracy on ImageNet-1K with 0.7 seconds on an iPhone 13. The star operation's ability to implicitly create high-dimensional feature spaces without additional computational overhead makes it a promising approach for efficient network design. The study also explores the star operation's potential in various scenarios, including networks without activation functions and different block designs. The results show that the star operation maintains strong performance even when activation functions are removed, and that the design of the star operation's blocks significantly impacts performance. The analysis highlights the star operation's unique ability to implicitly create high-dimensional feature spaces, which can be leveraged to improve network performance without increasing model complexity. The paper concludes that the star operation offers a novel approach to neural network design, with the potential to significantly improve efficiency and performance. The introduction of StarNet demonstrates the practical value of the star operation in real-world applications, and the study opens up new research directions in the field of efficient network design.This paper investigates the star operation (element-wise multiplication) in neural network design, revealing its ability to map inputs into high-dimensional, non-linear feature spaces without increasing network width. The authors demonstrate that the star operation can generate a feature space with approximately (d/√2)^2 linearly independent dimensions, similar to polynomial kernel functions. By stacking multiple layers, the star operation can exponentially increase the implicit feature dimensions, achieving nearly infinite dimensions within a compact network. This unique property makes the star operation highly efficient and suitable for compact network designs. The authors introduce StarNet, a simple and efficient network that leverages the star operation to achieve strong performance with minimal design complexity. StarNet outperforms several state-of-the-art efficient models in terms of accuracy and speed, achieving 73.5% top-1 accuracy on ImageNet-1K with 0.7 seconds on an iPhone 13. The star operation's ability to implicitly create high-dimensional feature spaces without additional computational overhead makes it a promising approach for efficient network design. The study also explores the star operation's potential in various scenarios, including networks without activation functions and different block designs. The results show that the star operation maintains strong performance even when activation functions are removed, and that the design of the star operation's blocks significantly impacts performance. The analysis highlights the star operation's unique ability to implicitly create high-dimensional feature spaces, which can be leveraged to improve network performance without increasing model complexity. The paper concludes that the star operation offers a novel approach to neural network design, with the potential to significantly improve efficiency and performance. The introduction of StarNet demonstrates the practical value of the star operation in real-world applications, and the study opens up new research directions in the field of efficient network design.
Reach us at info@study.space