Global Convergence of ADMM in Nonconvex Nonsmooth Optimization

Global Convergence of ADMM in Nonconvex Nonsmooth Optimization

May 31, 2018 | Yu Wang · Wotao Yin · Jinshan Zeng†
**Global Convergence of ADMM in Nonconvex Nonsmooth Optimization** This paper analyzes the convergence of the Alternating Direction Method of Multipliers (ADMM) for minimizing a nonconvex and possibly nonsmooth objective function, subject to coupled linear equality constraints. The ADMM updates each of the primal variables, followed by updating the dual variable. The variable y is separated from the x_i's due to its special role in the analysis. The developed convergence guarantee covers a variety of nonconvex functions, including piecewise linear functions, ℓ_q quasi-norm, Schatten-q quasi-norm, minimax concave penalty (MCP), and smoothly clipped absolute deviation (SCAD) penalty. It also allows nonconvex constraints such as compact manifolds and linear complementarity constraints. The x_0-block can be almost any lower semi-continuous function. The paper shows that several ADMM algorithms applied to solve nonconvex models in statistical learning, optimization on manifold, and matrix decomposition are guaranteed to converge. The results provide sufficient conditions for ADMM to converge on (convex or nonconvex) monotropic programs with three or more blocks. ADMM is compared to the augmented Lagrangian method (ALM). A simple example illustrates that ADMM converges but ALM diverges with bounded penalty parameter β. ADMM is shown to be more likely to converge for nonconvex nonsmooth problems due to its simplicity and robustness. The paper introduces a generalization of the coordinate descent method, Algorithm 1, which is proven to converge under certain assumptions. The algorithm is shown to reduce to the cyclic coordinate descent method when certain matrices are set to zero. The convergence of Algorithm 1 is established under assumptions on the objective function and matrices, including coercivity, feasibility, and Lipschitz sub-minimization paths. The paper also introduces new techniques for analyzing the convergence of ADMM, including an induction technique for nonconvex, nonsmooth cases, restricted prox-regularity, and more general linear mappings. These techniques allow the analysis of a wide range of nonconvex and nonsmooth functions, including piecewise linear functions, ℓ_q quasi-norms, and Schatten-q quasi-norms. The paper concludes that ADMM is a promising method for solving nonconvex nonsmooth optimization problems, with strong convergence guarantees and practical performance.**Global Convergence of ADMM in Nonconvex Nonsmooth Optimization** This paper analyzes the convergence of the Alternating Direction Method of Multipliers (ADMM) for minimizing a nonconvex and possibly nonsmooth objective function, subject to coupled linear equality constraints. The ADMM updates each of the primal variables, followed by updating the dual variable. The variable y is separated from the x_i's due to its special role in the analysis. The developed convergence guarantee covers a variety of nonconvex functions, including piecewise linear functions, ℓ_q quasi-norm, Schatten-q quasi-norm, minimax concave penalty (MCP), and smoothly clipped absolute deviation (SCAD) penalty. It also allows nonconvex constraints such as compact manifolds and linear complementarity constraints. The x_0-block can be almost any lower semi-continuous function. The paper shows that several ADMM algorithms applied to solve nonconvex models in statistical learning, optimization on manifold, and matrix decomposition are guaranteed to converge. The results provide sufficient conditions for ADMM to converge on (convex or nonconvex) monotropic programs with three or more blocks. ADMM is compared to the augmented Lagrangian method (ALM). A simple example illustrates that ADMM converges but ALM diverges with bounded penalty parameter β. ADMM is shown to be more likely to converge for nonconvex nonsmooth problems due to its simplicity and robustness. The paper introduces a generalization of the coordinate descent method, Algorithm 1, which is proven to converge under certain assumptions. The algorithm is shown to reduce to the cyclic coordinate descent method when certain matrices are set to zero. The convergence of Algorithm 1 is established under assumptions on the objective function and matrices, including coercivity, feasibility, and Lipschitz sub-minimization paths. The paper also introduces new techniques for analyzing the convergence of ADMM, including an induction technique for nonconvex, nonsmooth cases, restricted prox-regularity, and more general linear mappings. These techniques allow the analysis of a wide range of nonconvex and nonsmooth functions, including piecewise linear functions, ℓ_q quasi-norms, and Schatten-q quasi-norms. The paper concludes that ADMM is a promising method for solving nonconvex nonsmooth optimization problems, with strong convergence guarantees and practical performance.
Reach us at info@study.space
Understanding Global Convergence of ADMM in Nonconvex Nonsmooth Optimization