SNAS: STOCHASTIC NEURAL ARCHITECTURE SEARCH

SNAS: STOCHASTIC NEURAL ARCHITECTURE SEARCH

1 Apr 2020 | Sirui Xie, Hehui Zheng, Chunxiao Liu, Liang Lin
SNAS (Stochastic Neural Architecture Search) is an end-to-end solution for Neural Architecture Search (NAS) that trains neural operation parameters and architecture distribution parameters simultaneously during backpropagation, maintaining the completeness and differentiability of the NAS pipeline. Unlike traditional NAS methods, SNAS reformulates NAS as an optimization problem on a joint distribution of the search space, enabling efficient gradient-based learning. A novel search gradient is introduced, which optimizes the same objective as reinforcement-learning-based NAS but assigns credits to structural decisions more efficiently. This credit assignment is further augmented with locally decomposable rewards to enforce resource efficiency. SNAS outperforms non-differentiable evolution-based and reinforcement-learning-based NAS methods on CIFAR-10, achieving state-of-the-art accuracy with fewer epochs and transferable performance on ImageNet. SNAS also maintains validation accuracy during search, avoiding the need for parameter retraining required by attention-based NAS. The framework is efficient, interpretable, and less biased, making it a promising solution for efficient NAS on large datasets. SNAS uses a stochastic approach to architecture search, allowing for differentiable gradient updates and efficient resource constraints. The method is validated through experiments on CIFAR-10 and ImageNet, demonstrating its effectiveness in discovering high-performance convolutional cells with reduced parameters and improved accuracy. SNAS is shown to be more efficient than DARTS and ENAS, with the ability to further sparsify architectures under aggressive resource constraints. The framework is also transferable to larger datasets, showing competitive performance on ImageNet. SNAS is a novel, efficient, and highly automated framework for NAS, offering a more efficient and less biased alternative to existing methods.SNAS (Stochastic Neural Architecture Search) is an end-to-end solution for Neural Architecture Search (NAS) that trains neural operation parameters and architecture distribution parameters simultaneously during backpropagation, maintaining the completeness and differentiability of the NAS pipeline. Unlike traditional NAS methods, SNAS reformulates NAS as an optimization problem on a joint distribution of the search space, enabling efficient gradient-based learning. A novel search gradient is introduced, which optimizes the same objective as reinforcement-learning-based NAS but assigns credits to structural decisions more efficiently. This credit assignment is further augmented with locally decomposable rewards to enforce resource efficiency. SNAS outperforms non-differentiable evolution-based and reinforcement-learning-based NAS methods on CIFAR-10, achieving state-of-the-art accuracy with fewer epochs and transferable performance on ImageNet. SNAS also maintains validation accuracy during search, avoiding the need for parameter retraining required by attention-based NAS. The framework is efficient, interpretable, and less biased, making it a promising solution for efficient NAS on large datasets. SNAS uses a stochastic approach to architecture search, allowing for differentiable gradient updates and efficient resource constraints. The method is validated through experiments on CIFAR-10 and ImageNet, demonstrating its effectiveness in discovering high-performance convolutional cells with reduced parameters and improved accuracy. SNAS is shown to be more efficient than DARTS and ENAS, with the ability to further sparsify architectures under aggressive resource constraints. The framework is also transferable to larger datasets, showing competitive performance on ImageNet. SNAS is a novel, efficient, and highly automated framework for NAS, offering a more efficient and less biased alternative to existing methods.
Reach us at info@study.space
[slides] SNAS%3A Stochastic Neural Architecture Search | StudySpace