SHARPNESS-AWARE MINIMIZATION FOR EFFICIENTLY IMPROVING GENERALIZATION

SHARPNESS-AWARE MINIMIZATION FOR EFFICIENTLY IMPROVING GENERALIZATION

29 Apr 2021 | Pierre Foret, Ariel Kleiner, Hossein Mobahi, Behnam Neyshabur
The paper introduces Sharpness-Aware Minimization (SAM), a novel procedure that simultaneously minimizes both the training loss value and the loss sharpness to improve model generalization. SAM seeks parameters that lie in neighborhoods with uniformly low loss, which results in a min-max optimization problem that can be efficiently solved using gradient descent. Empirical results show that SAM improves generalization across various benchmark datasets and models, achieving state-of-the-art performance on several tasks. Additionally, SAM provides robustness to label noise, comparable to state-of-the-art procedures specifically designed for noisy labels. The paper also introduces the concept of *m-sharpness*, a measure of loss sharpness that is more correlated with generalization than the full-training-set measure. The authors discuss the theoretical and practical implications of SAM and suggest future directions for research.The paper introduces Sharpness-Aware Minimization (SAM), a novel procedure that simultaneously minimizes both the training loss value and the loss sharpness to improve model generalization. SAM seeks parameters that lie in neighborhoods with uniformly low loss, which results in a min-max optimization problem that can be efficiently solved using gradient descent. Empirical results show that SAM improves generalization across various benchmark datasets and models, achieving state-of-the-art performance on several tasks. Additionally, SAM provides robustness to label noise, comparable to state-of-the-art procedures specifically designed for noisy labels. The paper also introduces the concept of *m-sharpness*, a measure of loss sharpness that is more correlated with generalization than the full-training-set measure. The authors discuss the theoretical and practical implications of SAM and suggest future directions for research.
Reach us at info@study.space