A gradient-secant algorithm for unconstrained optimization problems is presented. The algorithm uses Armijo gradient method iterations until it reaches a region where the Newton method is more efficient, and then switches over to a secant form of operation. It is concluded that an efficient method for unconstrained minimization has been developed, and that any convergent minimization method can be substituted for the Armijo gradient method

Polak, E.

English

NASA Technical Reports Server

A MODIFIED SECANT METHOD FOR UNCONSTRAINED MINIMIZATION
by
E. Polak
Department of Electrical Engineering and Computer Sciences
and the Electronics Research Laboratory
University of California, Berkeley, California 94720
1. Introduction
The one variable secant method, for the solution of equations,
has been known for a very long time as being computationally more
efficient than Newton's method. Among the extensions of this method to
n-dimensional problems, those proposed by Wolfe [8 ] and Barnes [2 ],
are among the most interesting ones, because, when they do converge,
they ate emutpatktiaQn na &a-.si-t- ja sb ltic $hiF Ni-;¢'
method (see, for example, the discussion in [5 ]). However, as can be
seen from the counter examples quoted in [5 ], these methods may not
converge. More recently, Ritter [7 ] proposed a new algorithm for
function minimization, combining Goldstein's gradien m
with a secant type method, which contains an angl st (betPwenA
-\,=3
direction of descent and the gradient) to ensure ~Cnvert" e'e. hisir
angle test depends on a parameter that may be qui _ifDfi1'it to s ct
in advance. A bad selection results in the algorith,~. te
gradient mode most, if not all the time.
Research sponsored by the National Aeronautics and Space Administration
under Grant NGL-05-003-16, the National Science Foundation Grant GK-
10656X2, and the Joint Services Electronics Program, Contract
F-44620-71-C-0087.
fNA4SBA-CpCR1; 6 A %ODIFIED SECANT
IMETHOD FOR UNCONSTRAINED NIZATION73 0 6 3 3(California U niv) 17 p HC $3.00 I ATI N 3
CSCL 1 2A 1Ucas
G3/lq 17309
https://ntrs.nasa.gov/search.jsp?R=19730011906 2020-03-23T03:16:05+00:00Z
In this paper we present a new gradient-secant algorithm for
unconstrained optimization problems of the form min {f(z) I z E Rn}.
It differs from Ritter's method both in the fact that it does not use
an angle test and in the manner in which it updates the approximate
hessian. Roughly speaking, in solving a problem, this algorithm uses
Armijo gradient method iterations [ 1] until it reaches a region where the
Newton method is more efficient than the gradient method. Then it
switches over to a secant form of operation. Under the assumption that
f is continuously differentiable, we have shown that any accumulation
point z, of a sequence constructed by this algorithm, must be
stationary. Under the stronger hypothesis that f is twice continuously
differentiable and strictly convex, we were able to show that any
sequence {z.} constructed by our algorithm converges superlinearly
i=O
to the unique minimizer z of f(.), with rate Tz-, where T - is tne unique
n+1- n
positive root of t t - 1 = O, i.e. that for some 0e (0,1) and
i
some R E(0,), II zi - z II <R n , i=O, 1, 2, .. . Both theoretical
considerations and our computational experiments indicate that this new
algorithm is considerably faster than the Newton method, and Lootsma
[4]. reports that on many problems Newton's method is superior to
a number of conjugate direction and quasi-Newton methods. It is
therefore not unrealistic to hope that, as experience with the new method
accumulates, it will'emerge as one of the most efficient methods for
the solution of certain classes of unconstrained optimization problems.
-2-
2. The Secant Method.
Consider the problem
1. min{f(z) z E IRn }
To begin, we shall make only the following minimal assumptions.
2. Assumptions: (a) f is continuously differentiable and, (b) f is bounded
from below.
Throughout this paper, when we say that an algorithm is convergent,
we mean that every limit point z of a sequence it constructs in solving
(1) satisfies f(z) = 0.
The assumptions (2) will suffice to prove that the algorithm we are
about to. state is convergent. We shall later show, under stronger
assumptions, that it converges superlinearly and establish a bound on
We shall use the notation
3. g(z) E Vf(z), z e IRn'.
4. Algorithm:
Data: 6 > 0, a E (0, ),  E (0,1), b > 0 (large)' ,g > 2,z0 IR
th
H a symmetric positive definite n x n matrix,ej = j- column of n x n
unit matrix, j = 1, 2, .., n.
Step 0: Set i = 0, j = 0, p = 0, v0 = 6, H  H. Compute g(z0) and
set Y0 = v0 = j g(z0 ) 2'
Step 1: Compute g(zi). Stop if g(zi) = 0.
Step 2: If j < n, set'j = j + 1 and go to step 3; else-set j = 1 and
t The purpose of the constant b is to make the algorithm use a steepest
descent step whenever Hi, the current approximation to the hessian of f(-)
is "too close" to being singular. A lower bound on b is b > 2 JJ H(z)-
for all z which are local minimizers of f(-). In practice, setting
b = c does not appear to destroy the convergence of the algorithm.
-3-
go to step 3.
Step 3: Set ei = min{6,vi}.
Step 4: Compute g(zi + ei e.).
Step 5: Replace hj, the jt column of H, by
5. A. =- [g(zi + E ej) - g(zi)]1. igizi1
to obtain a new matrix
Step 6: If IIg(zi) II <
Step 7: If H1 exists
1
H, and set H. = H.
1,
p , go the step 7; else set W = zi go to step 15.
and II H 1 il < b, compute
1
6. v.= H g(zi)
and go to step 8; else set w = z. and go to step 15.
Step 8: If (v., g(z i) ) < 0, go to step 9; else set w = z.
Step 9: Set k = 0.
Step 10: Compute f(z i - 8kv.).
Step 11: If
and go to
7. f(zi -k v.) - fkzi) < 0
go to step 13; else go to step 12.
Step 12: If k < 2, set k = k + 1 and go to step 10; else go. to step 15.
Step 13: Compute g(Zi - ). If g(i ) = 0,
set zi+ = - vi and stop.i+l l 1
Step 14: If
k2
set zi+ = - set = l g(zi+l) 2, set p = p+l, set i = i-+
-1 -1
t Note that since H differs from H in only one column,H. = H -1
new old 1 new
can be obtained from H-1od by means of the standard updating formula.
Hold
-4-
and go to step 2; else set w = zi - 8 vi and go to step 15.
Step 15: Compute the smallest integer s .> 0 such that1-
S. S.
9. f(zi -8 1 g(zi)) - f(zi) < - II g(zi) II
S.
and set y = z. - 8 g(zi).
Step 16: If f(y) < f(w), set zi+l = y, set i = i+l, and go to step 1;
else set z = w, set i = i+l and go to step . ni+l
Since when g(zi ) # 0, one can always find a finite s. such that (9)
is satisfied, algorithm (4) is obviously well defined.
10. Theorem: Suppose that the assumptions (2) are satisfied and that
algorithm (4) has constructed an infinite sequence {zi}i=O. Then every
limit point z* of {zi i=0 satisfies g(z*) = 0.
Proof: Suppose that z* is a limit point of {zi}, that g(z*) # 0 and that
Now there are two possibilities.
(i) There exists an infinite subset K' C K such that for all i E K',
either
S .
11. Z = z - 1 g(zi)
or
zi+l = Zi - k v.i+l 1 1
12.
and
13. f(zi - S vi) < f(zi - 8 ig(zi).
Since g(z*) i 0 and z. - z* for i C K', it follows from the discussion in
1
.-5-
sec. 2.1 of [ 6] (see theorem 22 and algorithm 35) that there exists
a 6(z*) < 0 and an integer N such that
14. f(z i+l) - f(zi ) < 6 (z*) < 0 for all i > N, i E K'.
But K' is an infinite subset, and {f(zi)}i= is a monotonically
decreasing sequence, hence (14) contradicts the assumption that f(z) is
bounded from below. Thus, if (i) holds, then g(z*) = 0.
The second possibility is
(ii) There exists an infinite subset K" C K such that
15. Zi+1 z. - kv for all i E K"
and
16. Ij g(zi+l) 2 < (1-28 a) g (z) Ij for all i E K"
In this case the sequence {yp is infinite, monotonically decreasing and
bounded from below by zero. Hence y + y* > O. Now, since whenever (15)
and (16) take place, y = .II g(zi+) 112, for some integer p, and
p-I i+l
y > I g(z ) 2 'y < g 112 < - g(z) I'2. Hence, since
%ll ¥- Yp+llj g(zi)j
z. - z* for i E-K", and since g(-) is continuous by assumption (2),
there exists an infinite subset K"' of the positive integers such that
17. Yp+l - Yp < -- B a I g(z*) II2 for all p E K".
But (17) contradicts the fact that yp + y* > O.
Hence we must have g(z*) = O. n
The following result is a direct consequence of theorem (10).
18. Corollary: Algorithm (4) is convergent whenever problem (1) satisfies
-6-
the assumptions (2),
The following corollary can be deduced from theorem (10), which
implies that g(zi ) ~ 0 as i - ? and hence that zi+l Zi 
* 0 as i 9 I,
and theorem (1.3.66) in [ 6].
19. Corollary: Suppose that the sequence {z I described in theorem
(10) is compact and that the function f(,) has only a finite number
of stationary points, then there exists a z* E Rn such that zi Z z*
and g(z*) 0. 
We are now ready to establish the rate of convergence of algorithm
(4). For this purpose we shall need to assume the following.
20. Assumption: The function f is (a) three times continuously differentiable
and, (b) strictly convex.
Note that under assumption (20), the level sets of f(') are
compact and there exists only one point z ( the minimizer of f(z)
over II ), which satisfies g(z) O. Hence., by theorem (10) and
corollary (19), whenever assumption (20) is satisfied, any sequence
{zi}a constructed by algorithm (4) converges to the unique mini-
mizer z of f(-).
21. Lemma: Suppose that assumption (20) is satisfied and that algorithm (4)
has constructed an infinite sequence {zii0 converging to z, the
minimizer of f(.), Then there exists 0 < M < such that
i-n
22. II H(zi) i Hi 1 < M z II for i 0= , 1, 2,
where Hi is as defined in the algorithm and
2
23. H(z) a z E R
az
Proof: Since (20) (b) is satisfied, the level set
C(z) = {z I f(z) < f(z )} is compact and convex and hence, since (20) (a) is
satisfied, there exists a Lipshitz constant L < ~ such that for all
x, y E C(z),
24. II. H(x) - H(y) II < L II x - y II
(Note that zi}i=0 is contained in C(z0 )). Now, without loss of
generality, suppose that the jth column of Hi(j E {1, 2, ..., n}) is
25. i [g(zi k + Ei-k ej) - g(zik)] where k E {0, 1, 2, .. , n-l}.Ci-k i- ' ' ''
Then, making use of (24), of the mean value theorem, and the fact that
ik < ki Zi-k-l II by construction, we obtain that the magnitude
th
of the difference between the jth columns of H(zi) and H. satisfies
26. I H(zi) e - + c ej) - g(z zi )]
1 EiHk i-k i-k i-k
= II f [H(zi) - H(z + ti e)] e. dt IIi-k i-k
0
II Zi- ik t II dt
1
_Zi z-k zik i-k-l )
0
-8-
< L(1 z-zI I + 3 II z II + 1 IIik-z ll)
-L( i 2. -51 Zi-k 2 i-k-1 
The existence of a constant M satisfying (22) now follows from (26)
and the triangle inequality for norms, used in conjunction with the
addition and subtraction of terms in the right hand side of (6). n
27. Lemma: Suppose that assumption (20) is satisfied, that b > 2 II H(z) 1
and that the algorithm (4) has constructed a sequence {zi}i =
Then there exists an integer N such that for all i > N,
z i + H= H- 1
i+l. Zi - i g(z i) .
Proof: First, since zi - z, the global minimizer of f(-), and (22)
holds, it follows from the perturbation Lemma (2.3.2) in [ 5] that
there exists an integer N' such that for all i > N' H
-
exists and is
positive definite and II H
-
1 il < b. Hence, for all i > N'. the test in
step 8 of the algorithm, i.e., (vi, g(zi) )= -<(l g(zi), g(zi ) ) < 0,
is satisfied for all i > N' and hence the computation proceeds to step 9.
Next, applying the second order Taylor expansion, we obtain (with
-1
vi = Hi g(zi))
28. f(zi Hi g(zi)) f(zi)
- 1
H -1
-(g(zi), H i g(z i ) ) + (l-t) [(H g(1 z H(z) H g(z.) 
+('g~z ), H g(zi)' H(zi-tvi }til g(zi) ) dt
1.-( i )- 'i) i i l H1(g= z H-1 g~z ((-t) H. g(z i H( ) H- g(z.) )
' 
~
-9-
+ (Hi g(zi),(H(zi -t vi) - H(z)) Hi g(zi ) )] dt
-1 -1 g i
Hence, since vi = H
-
g(zi) + 0 as i + o (because H 1- H(z)
-
1 1 ' 1
and g(zi ) + 0 causing H(zi - vi) - H(z) + O uniformly for t E [0,1]
as i + 0), and since H(z) H
-
-+ I, the unit matrix, as i + A, it follows
1
from (28) that there is an integer N" > N' such that
29. f(z Hi g(zi)) - f(zi) < 0 for all i > N",
i.e. the test in step 11 of the algorithm is satisfied with k = 0 for all
i > N".
Finally, consider 2 g(z) 2. Since f is three times continuously
differentiable,
2
30. (z) = H(z.) H(z) + W(z)
az' =
where W(z) is a continuous nxn matrix which satisfies W(z) = 0. Hence,
for all i > N", expanding I1 g(zi-H.i g(zi))l2 to second order terms
according to the Taylor formula, we obtain
31. II g(zi -H. 1 g2 I (zi 2 2 < H(zT g(zi) i)) )g(zi-H g(z.))I i) }
1
H-1 -l
+ 2 |(l-t)[<Hi g(zi), H(zi i i-tv ) Hi g(zi) )
1 g(zi)' Hi g(zi ) 
Where v. = H
-
g(zi). Setting Hi(t) = H(zi-tvi) and Wi(t) = W(zl-tvi),
t E'[0,l], (31) yields
-10-
32. i ~z 2 g ,2 - ( g(zi)' H(zi) H:1 g(
z
i
) )32. (zivi 2 II g(zi) II2 -2 1
II'pg(zi) 112
1
+ (-t i(t) Hi 1 g(zi)I 2 +
0
(Hi g(zi), Wi(t) Hi g(zi ) dt
Since v. + 0 as i + m and z. + z as i + A, Wi(t) + 0 as i + A,
uniformly in t E [0,1] and similarly, Hi(t) Hi -+ I as i + o, uniformly in
1
t E [0,1]. Hence the term in the right hand side of (32), multiplied
by 1 g(zi) 112 tends to zero as i + X and therefore there exists an integer
N"'> N" such that the test (8) is satisfied for k = 0 for all i > N"'.
Now, since g(zi) + 0 as i + w, there exists an integer N > N"' at
which the test in step 6, viz.,11g(zi) 112 < p will be satisfied. Then,
for all i > N, zi+l = Zi - Hi g(Zi) which completes our proof. M
33. Theorem: Suppose that assumption (20) is satisfied and that algorithm
co
(4) has constructed a sequence {zi}i = 0 Then
1/i i=i
34. 0 < lim sup z. - z II <
i-to
where T is the unique positive root of the equation t t - 1 0
n
and z is the unique minimizer of f(-) (i.e., the R-order of algorithm (4)
is Tn, where R-order is defined by (9.2.5) in [ 5]).
Proof: Let N be an integer such that for all i > N
-11-
-1
35. zi+l = Zi - Hi g(zi )
By Lemma (27), such an N exists. Then since H(-) is Lipshitz
continuous on C(z0 ), for all i > N (since g(z) = 0)
36. Zi+l z II = II(zi-z) - Hi (g(zi)-g(z))1
1
0 II
0
(I-H H( t ( ) (1IH 1d- 1Zi ))(i)d~
II H
-
1 II I(Hi-H(z+t(zi-z ))Il Iz. - z Ildt
1
11 -1 
< II H-1 II
i-- 1
1
(ri Hi-.-HC(7A II + II H(z,) - H(z+t(..--))l 1 liz.-zl ) dt
o
3_ 1 ' 1 1 2 1
0
1 
[lHi-H(zi) II + Lt II z.-z ] z- Ildt'
O
where L is the Lipshitz constant for H(-) on C(z0 ). Now making use of
Lemma (22) and the fact that Ii H-1 II is bounded for i> N
(since H1 + H(z)), we conclude from (36) that there exists constants
1
X > 0, j = 0; 1, 2, .., n-l, such that for all i > Nj
n
37. 11 zi+l - 1 < i11 z - z I2 A 11 z.=
j=o
- II..
-12-
The desired result now follows from ( 7) and theorem (9.2.9) in [5 ].
38. Corollary: Under the conditions in theorem (33), any sequence {z } i=
constructed by algorithm (4) satisfies
n
39. lim sup I g(zi) n < 
i-i~o/
1/T i
40. lim sup [f(zi ) - f(z)] < 1
i-do
Proof: Since g(z) = 0,
1
41. II g(zi) I = II H(z +t(zi-z )) (zi-z) dt II
v
1 1
--< [ f I IHt(z+t(zi- II dt]t[ zi.- z i
0
< Q II zi-z I
where Q = sup {l1 H(z) II | z E C(z0)}. Relation (39) now follows from
1/T
(41), (34) and the fact that Q + 1 as i A.
Next, again since g(z) = 0,
-13-
142. f(z) - f(z) (l-t)( (Z i-z),H(z+t(z -z)) ( )) dt
0
1
(l-t) Q i z. - z II dt
0
2 .
where Q is an upper bound on the eigenvalues of H(z) for z G C(z0 ).
1/2T
Relation (40) now follows from (42), (34) and the fact that (- ) -+ 1
as i + a. This completes our proof. n
we lottd"tfL-L Llhe 2i y Li6 we ii us_ - thF\
was three times continuously differentiable and strictly convex was in
the proof of lemma (27). At this point is is easy to show that lemma (27)
can also be proved under the weaker assumption that f(.) is only twice
continuously differentiable strictly convex and (24) holds. Thus suppose that
{zi}i = is any sequence such that z. + z as i + X and that
Yi- = Zi - Hi! g(zi) for all i > N, where N is such that H. exists
for all i > N. Then (37) applies and yields
n
'~43.- I - zy11 Z^. - z 1 11z zl j - z .i, for all I > Ni+l z i z
j=O
Now, by the mean value theorem, since g(z) = 0,
1
44. 11 g(y +l ) II < II H(yil +s(Yi1 - z)l ds 1 Yi+l Z 
0
-14-
< Q I Ai
-<QIIYi+l
- Z
n
< Q I] z.-z I Jl zi-j-z IZ.
- 1-3
j=0
where Q is as in (41).
Now,
45. U z.-z H II g(zi) > (z.-z, g(zi))
(-z, 'H(z +s (z.-z) (z -z))ds
O'
> Q II z.-z I[2 ,
where Q is as in (42). Hence for all i > N
46. 1 g(zi) > I z-z 
and therefore (44) yields
n
47. g ( {Q <l ziZj-z II} for all i > 
j=0
n
Since !l z._z ll 0 as i + ~ , we conclude that there exists an
j=0
integer N > N such that
1I g(yi+l) II < (1-2cB ) II g(zi) I2 for all i > N
-15-
and hence that lemma (27) holds under the weaker assumption that f is
only twice continuously differentiable and strictly convex.
Conclusion
We have presented in this paper an efficient method for unconstrained
minimization. It should be clear from the development that the
assumptions used to establish rate of convergence can be relaxed from a
global statement to a local one, i.e.' as holding in a convex neighborhood
of a local minimum. It is also clear that one can construct several
other variants of the algorithm as, for example, by substituting a
conjugate directions method for the gradient method in the algorithm.
In some applications these alternative, more complex versions may be
preferred over the simplest one presented in this paper. -As long as one
.. ti..Ui .... fc tha ArmiJo gradiAnt mathod anly oihehr codvezgaiiL ii-
zation method, the convergence and rate of convergence theorems, presented
in this paper, remain valid.
-16-
REFERENCES
[11 L. Armijo, "Minimization of functions having continuous partial
derivatives," Pacific J. Math. 16, 1-3 (1966).
[2] J. Barnes, "An algorithm for solving nonlinear equations based on
the secant method," Comp. J., 8, 66-72 (1965).
[3] A. A. Goldstein, "Constructive Real Analysis, Harper and Rowe, New
York, 1967, Ch. 1, Section D.
[4] F. A. Lootsma, "Penalty function performance of several unconstrained-
minimization techniques," Philips Res. Repts. 27, 358-385 (1972).
[5] J. M. Ortega and W. C. Rheinboldt, Iterative Solution of Nonlinear
Equations in Several Variables, Academic Press, New York, 1971.
[6] E. Polak, Computational Methods in Optimization, Academic Press,
New York, 1971.
[7] K. Ritter, "A Superlinearly convergent method for unconstrained
minimization," in Nonlinear Programming, J. B. Rosen, O. L.
Mangasarian and K. Ritter, Eds., Academic Press, New York 1970,
pp. 177-206.
[8] P.. Wolfe, The secant method for simultaneous nonlinear equations,
Comm. ACM, 2, 12-14, 1959.
-17-


A modified secant method for unconstrained minimization

Abstract

Similar works

Full text

Available Versions

NASA Technical Reports Server