The Expressive Power of Neural Networks: A View from the Width

The Expressive Power of Neural Networks: A View from the Width

1 Nov 2017 | Zhou Lu, Hongming Pu, Feicheng Wang, Zhiqiang Hu, Liwei Wang
This paper investigates the expressive power of neural networks from the perspective of width, contrasting with previous studies that focus on depth. We prove a universal approximation theorem for width-bounded ReLU networks, showing that width-(n+4) ReLU networks can approximate any Lebesgue-integrable function on n-dimensional space with respect to L¹ distance. Additionally, we demonstrate that except for a measure zero set, all functions cannot be approximated by width-n ReLU networks, indicating a phase transition in expressive power. We also explore the width efficiency of ReLU networks, showing that there exist classes of wide networks that cannot be realized by narrow networks with depth no more than a polynomial bound. Experimental results indicate that narrow networks with size slightly larger than the polynomial bound can approximate wide and shallow networks with high accuracy. Our findings suggest that depth may be more effective than width for the expressive power of ReLU networks. We also pose open problems regarding the width efficiency of ReLU networks, including whether an exponential lower bound or polynomial upper bound holds. The results provide more comprehensive evidence that depth is more effective than width for the expressive power of ReLU networks.This paper investigates the expressive power of neural networks from the perspective of width, contrasting with previous studies that focus on depth. We prove a universal approximation theorem for width-bounded ReLU networks, showing that width-(n+4) ReLU networks can approximate any Lebesgue-integrable function on n-dimensional space with respect to L¹ distance. Additionally, we demonstrate that except for a measure zero set, all functions cannot be approximated by width-n ReLU networks, indicating a phase transition in expressive power. We also explore the width efficiency of ReLU networks, showing that there exist classes of wide networks that cannot be realized by narrow networks with depth no more than a polynomial bound. Experimental results indicate that narrow networks with size slightly larger than the polynomial bound can approximate wide and shallow networks with high accuracy. Our findings suggest that depth may be more effective than width for the expressive power of ReLU networks. We also pose open problems regarding the width efficiency of ReLU networks, including whether an exponential lower bound or polynomial upper bound holds. The results provide more comprehensive evidence that depth is more effective than width for the expressive power of ReLU networks.
Reach us at info@study.space