The paper "The Marginal Value of Adaptive Gradient Methods in Machine Learning" by Ashia C. Wilson, Rebecca Roelofs, Mitchell Stern, Nathan Srebro, and Benjamin Recht explores the performance of adaptive optimization methods in machine learning, particularly in deep neural networks. Adaptive methods, such as AdaGrad, RMSProp, and Adam, are popular due to their rapid training speed but have been shown to find solutions that generalize poorly compared to non-adaptive methods like stochastic gradient descent (SGD) and its variants. The authors construct a binary classification problem where the data is linearly separable. They demonstrate that while SGD and its variants achieve zero test error, adaptive methods like AdaGrad, Adam, and RMSProp can attain test errors arbitrarily close to half. This suggests that adaptive methods tend to give undue influence to spurious features, leading to poor generalization. Empirical studies on several state-of-the-art deep learning models further support these findings. The results show that adaptive methods generally generalize worse than non-adaptive methods, even when they achieve the same training loss or lower. Additionally, adaptive methods often display faster initial progress but quickly plateau on the development or test set. The paper also highlights the importance of hyperparameter tuning, particularly for learning rates and decay schemes, which can significantly improve the performance of adaptive methods. The authors conclude that practitioners should reconsider the use of adaptive methods for training neural networks, as they may not provide the same level of generalization as non-adaptive methods. They propose a simple scheme for tuning learning rates and decays that performs well across various deep learning tasks.The paper "The Marginal Value of Adaptive Gradient Methods in Machine Learning" by Ashia C. Wilson, Rebecca Roelofs, Mitchell Stern, Nathan Srebro, and Benjamin Recht explores the performance of adaptive optimization methods in machine learning, particularly in deep neural networks. Adaptive methods, such as AdaGrad, RMSProp, and Adam, are popular due to their rapid training speed but have been shown to find solutions that generalize poorly compared to non-adaptive methods like stochastic gradient descent (SGD) and its variants. The authors construct a binary classification problem where the data is linearly separable. They demonstrate that while SGD and its variants achieve zero test error, adaptive methods like AdaGrad, Adam, and RMSProp can attain test errors arbitrarily close to half. This suggests that adaptive methods tend to give undue influence to spurious features, leading to poor generalization. Empirical studies on several state-of-the-art deep learning models further support these findings. The results show that adaptive methods generally generalize worse than non-adaptive methods, even when they achieve the same training loss or lower. Additionally, adaptive methods often display faster initial progress but quickly plateau on the development or test set. The paper also highlights the importance of hyperparameter tuning, particularly for learning rates and decay schemes, which can significantly improve the performance of adaptive methods. The authors conclude that practitioners should reconsider the use of adaptive methods for training neural networks, as they may not provide the same level of generalization as non-adaptive methods. They propose a simple scheme for tuning learning rates and decays that performs well across various deep learning tasks.