This paper introduces a new network design paradigm that focuses on designing network design spaces rather than individual network instances. The goal is to discover general design principles that apply across various settings. The proposed method, RegNet, parametrizes populations of networks using a simple, quantized linear function to determine widths and depths. This approach leads to a low-dimensional design space consisting of simple, regular networks that perform well across a wide range of flop regimes. RegNet models outperform popular models like EfficientNet while being up to 5× faster on GPUs under comparable training settings. The design space design process involves progressively simplifying an initial, unconstrained design space while maintaining or improving its quality. This process is analogous to manual network design but elevated to the population level. The RegNet design space is shown to generalize well across different compute regimes, schedule lengths, and network block types. The paper also compares RegNet models to existing networks in various settings, demonstrating their effectiveness in the mobile regime and across different flop regimes. The results show that RegNet models provide significant improvements over standard ResNe(X)T models and outperform EfficientNet models in terms of performance and speed. The paper concludes that designing network design spaces is a promising avenue for future research.This paper introduces a new network design paradigm that focuses on designing network design spaces rather than individual network instances. The goal is to discover general design principles that apply across various settings. The proposed method, RegNet, parametrizes populations of networks using a simple, quantized linear function to determine widths and depths. This approach leads to a low-dimensional design space consisting of simple, regular networks that perform well across a wide range of flop regimes. RegNet models outperform popular models like EfficientNet while being up to 5× faster on GPUs under comparable training settings. The design space design process involves progressively simplifying an initial, unconstrained design space while maintaining or improving its quality. This process is analogous to manual network design but elevated to the population level. The RegNet design space is shown to generalize well across different compute regimes, schedule lengths, and network block types. The paper also compares RegNet models to existing networks in various settings, demonstrating their effectiveness in the mobile regime and across different flop regimes. The results show that RegNet models provide significant improvements over standard ResNe(X)T models and outperform EfficientNet models in terms of performance and speed. The paper concludes that designing network design spaces is a promising avenue for future research.