February 1992 | JEAN CHARLES GILBERT AND JORGE NOCEDAL
This paper investigates the global convergence of nonlinear conjugate gradient methods without restarts and with practical line searches. It analyzes two classes of methods that are globally convergent on smooth, nonconvex functions. The Fletcher–Reeves method plays a key role in the first class, while the second class shares a property with the Polak–Ribière method. Numerical experiments are presented.
Key words: conjugate gradient method, global convergence, unconstrained optimization, large-scale optimization
The paper studies the convergence of conjugate gradient methods for nonlinear optimization. It considers methods without regular restarts and asks under what conditions they are globally convergent for general smooth nonlinear functions. The analysis highlights differences among various conjugate gradient methods and suggests new implementations.
The problem is to minimize a function of n variables, $ \min f(x) $, where $ f $ is smooth and its gradient $ g $ is available. The iterations are of the form:
$$ d_{k}=\left\{\begin{array}{l l}{-g_{k}}&{\mathrm{f o r}k=1,}\\ {-g_{k}+\beta_{k}d_{k-1}}&{\mathrm{f o r}k\geq2,}\end{array}\right. $$
$$ x_{k+1}=x_{k}+\alpha_{k}d_{k}, $$
where $ \beta_{k} $ is a scalar, and $ \alpha_{k} $ is a steplength obtained by means of a one-dimensional search. The paper discusses the Fletcher–Reeves (FR), Polak–Ribière (PR), and Hestenes–Stiefel (HS) formulas for $ \beta_{k} $, and their numerical performance.
The FR method is sometimes as efficient as PR and HS, but often slower. Powell showed that the FR method with exact line searches can produce very small displacements and not recover unless restarted. Zoutendijk proved that the FR method with exact line searches is globally convergent on general functions. Al-Baali extended this result to inexact line searches.
The PR and HS methods perform similarly in practice and are preferred over FR. However, Powell showed that the PR method with exact line searches can cycle infinitely without approaching a solution. The same result applies to the HS method. Al-Baali's convergence result for the FR method is very satisfactory.
The paper considers various choices of $ \beta_{k} $ and line search strategies that result in globally convergent methods. It establishes global convergence for methods with $ |\beta_{k}| \leq \beta_{k}^{FR} $ and describes a modification of the PR formula. It also considers methods with nonnegative $ \beta_{k} $, and shows that setting $ \beta_{k} = \max\{\beta_{k}^{PR}, 0\}This paper investigates the global convergence of nonlinear conjugate gradient methods without restarts and with practical line searches. It analyzes two classes of methods that are globally convergent on smooth, nonconvex functions. The Fletcher–Reeves method plays a key role in the first class, while the second class shares a property with the Polak–Ribière method. Numerical experiments are presented.
Key words: conjugate gradient method, global convergence, unconstrained optimization, large-scale optimization
The paper studies the convergence of conjugate gradient methods for nonlinear optimization. It considers methods without regular restarts and asks under what conditions they are globally convergent for general smooth nonlinear functions. The analysis highlights differences among various conjugate gradient methods and suggests new implementations.
The problem is to minimize a function of n variables, $ \min f(x) $, where $ f $ is smooth and its gradient $ g $ is available. The iterations are of the form:
$$ d_{k}=\left\{\begin{array}{l l}{-g_{k}}&{\mathrm{f o r}k=1,}\\ {-g_{k}+\beta_{k}d_{k-1}}&{\mathrm{f o r}k\geq2,}\end{array}\right. $$
$$ x_{k+1}=x_{k}+\alpha_{k}d_{k}, $$
where $ \beta_{k} $ is a scalar, and $ \alpha_{k} $ is a steplength obtained by means of a one-dimensional search. The paper discusses the Fletcher–Reeves (FR), Polak–Ribière (PR), and Hestenes–Stiefel (HS) formulas for $ \beta_{k} $, and their numerical performance.
The FR method is sometimes as efficient as PR and HS, but often slower. Powell showed that the FR method with exact line searches can produce very small displacements and not recover unless restarted. Zoutendijk proved that the FR method with exact line searches is globally convergent on general functions. Al-Baali extended this result to inexact line searches.
The PR and HS methods perform similarly in practice and are preferred over FR. However, Powell showed that the PR method with exact line searches can cycle infinitely without approaching a solution. The same result applies to the HS method. Al-Baali's convergence result for the FR method is very satisfactory.
The paper considers various choices of $ \beta_{k} $ and line search strategies that result in globally convergent methods. It establishes global convergence for methods with $ |\beta_{k}| \leq \beta_{k}^{FR} $ and describes a modification of the PR formula. It also considers methods with nonnegative $ \beta_{k} $, and shows that setting $ \beta_{k} = \max\{\beta_{k}^{PR}, 0\}