SHARPNESS-AWARE MINIMIZATION FOR EFFICIENTLY IMPROVING GENERALIZATION

SHARPNESS-AWARE MINIMIZATION FOR EFFICIENTLY IMPROVING GENERALIZATION

29 Apr 2021 | Pierre Foret, Ariel Kleiner, Hossein Mobahi, Behnam Neyshabur
**Sharpness-Aware Minimization for Efficiently Improving Generalization** This paper introduces Sharpness-Aware Minimization (SAM), a novel optimization method that improves model generalization by simultaneously minimizing both the training loss and the sharpness of the loss landscape. SAM seeks parameters that lie in neighborhoods with uniformly low loss, leading to a min-max optimization problem that can be efficiently solved with gradient descent. The method is effective across various benchmark datasets and models, achieving state-of-the-art performance in tasks such as image classification and robustness to label noise. SAM also provides robustness to label noise comparable to state-of-the-art methods specifically designed for noisy label learning. The paper presents empirical results showing that SAM improves generalization across a wide range of tasks, including CIFAR-10, CIFAR-100, ImageNet, and finetuning tasks. Additionally, SAM introduces a new notion of sharpness, m-sharpness, which better captures the relationship between loss sharpness and generalization. The method is implemented efficiently and is compatible with existing training techniques, making it a powerful complement to current approaches in deep learning.**Sharpness-Aware Minimization for Efficiently Improving Generalization** This paper introduces Sharpness-Aware Minimization (SAM), a novel optimization method that improves model generalization by simultaneously minimizing both the training loss and the sharpness of the loss landscape. SAM seeks parameters that lie in neighborhoods with uniformly low loss, leading to a min-max optimization problem that can be efficiently solved with gradient descent. The method is effective across various benchmark datasets and models, achieving state-of-the-art performance in tasks such as image classification and robustness to label noise. SAM also provides robustness to label noise comparable to state-of-the-art methods specifically designed for noisy label learning. The paper presents empirical results showing that SAM improves generalization across a wide range of tasks, including CIFAR-10, CIFAR-100, ImageNet, and finetuning tasks. Additionally, SAM introduces a new notion of sharpness, m-sharpness, which better captures the relationship between loss sharpness and generalization. The method is implemented efficiently and is compatible with existing training techniques, making it a powerful complement to current approaches in deep learning.
Reach us at info@study.space