Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Condition

Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Condition

12 Sep 2020 | Hamed Karimi, Julie Nutini, and Mark Schmidt
This paper investigates the linear convergence of gradient and proximal-gradient methods under the Polyak-Lojasiewicz (PL) condition. The PL condition is a sufficient condition for global linear convergence of gradient descent, and it is a special case of the Lojasiewicz inequality. The authors show that the PL condition is weaker than other conditions used to achieve linear convergence without strong convexity. They use the PL condition to analyze various optimization methods, including randomized and greedy coordinate descent, sign-based gradient descent, and stochastic gradient methods. They also propose a generalization of the PL condition for non-smooth optimization, leading to simple proofs of linear convergence for proximal-gradient methods. The paper also shows that the PL condition applies to a wide range of machine learning problems, including least squares, logistic regression, boosting, and support vector machines. The authors conclude that the PL condition provides a unifying and simplifying view of optimization and convergence rate issues in machine learning.This paper investigates the linear convergence of gradient and proximal-gradient methods under the Polyak-Lojasiewicz (PL) condition. The PL condition is a sufficient condition for global linear convergence of gradient descent, and it is a special case of the Lojasiewicz inequality. The authors show that the PL condition is weaker than other conditions used to achieve linear convergence without strong convexity. They use the PL condition to analyze various optimization methods, including randomized and greedy coordinate descent, sign-based gradient descent, and stochastic gradient methods. They also propose a generalization of the PL condition for non-smooth optimization, leading to simple proofs of linear convergence for proximal-gradient methods. The paper also shows that the PL condition applies to a wide range of machine learning problems, including least squares, logistic regression, boosting, and support vector machines. The authors conclude that the PL condition provides a unifying and simplifying view of optimization and convergence rate issues in machine learning.
Reach us at info@study.space
[slides] Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-%C5%81ojasiewicz Condition | StudySpace