Quadratically convergent gradient methods for minimizing unconstrained function of several variable

Myers, G. E.

English

NASA Technical Reports Server

General Disclaimer 
One or more of the Following Statements may affect this Document 
 
 This document has been reproduced from the best copy furnished by the 
organizational source. It is being released in the interest of making available as 
much information as possible. 
 
 This document may contain data, which exceeds the sheet parameters. It was 
furnished in this condition by the organizational source and is the best copy 
available. 
 
 This document may contain tone-on-tone or color graphs, charts and/or pictures, 
which have been reproduced in black and white. 
 
 This document is paginated as submitted by the original source. 
 
 Portions of this document are not fully legible due to the historical nature of some 
of the material. However, it is the best reproduction available from the original 
submission. 
 
 
 
 
 
 
 
Produced by the NASA Center for Aerospace Information (CASI) 
https://ntrs.nasa.gov/search.jsp?R=19680014586 2020-03-12T10:37:28+00:00Z
of	 {
nlastq cR-920?
11 1,;11 It poll
1'A RT V
"I'TiOl 3 fe R'1'II S OT'' 'I'lC1^; GC)I^JiT ,A'L'1'GRADIENT   D
Genaldine E, y 1^
GPO PRICE	 $
CFSTI PRICE(S) $
	
Hard copy (HQ	
>>
	
Microfiche (MF)	 (D
H 653 July 65
Repor t No. 68-8
Conti-act. NAS 9--40,36
Awl1 1 968
..''(ACCEUNM NU	 )	 (THRU)
Cx? Ll
^.^	 (SAGES	 (CODE)
(NASA CR OR TMX OR AD NUM6E	 (CATEGORY)
,&' A_f.VTI;C,A[. M.ECIIANIC:S ASS40 %I.A.T.ES, ,IN(`.
r 57 01.1) G011N'11tY 13OAII
%VVNI'M J IIY, N. Y. 1151111
k-
w I
`t	 1
10AS^ e"006 9 a la,,9
z
PROPERTIES OF THE CONJUGATE GRADIENT AND
DAVIDON METHODS*
Geraldine E. Myers**
Analytical Mechanics Associates, Inc.
Westbury, New York
ABSTRACT
Two quadraticsIly convergent gradient methods for minimizing an uncon-
strained function of several variables are examined. The heart of the Fletcher
and Powell reformulation of Davidon' s method is a variable H matrix. The
eigenvalues and eigenvectors of this matrix for a quadratic function are explored,
leading to a proof that the gradient vectors at each step are mutually orthogonal.
From this is derived a geometric interpretation of the H matrix in terms of the
projection of the gradient into a solution subspace.
These properties are used to arrive at the main result, which states that,
for a quadratic function, the direction vectors generated by the Davidon algorithm
and the conjugate gradient algorithm of Hestenes and Stiefel are scalar multiples
of each other, provided the initial step each takes is in the direction of steepest
descent. Assuming no round=off and a perfect one-dimensional search, the
methods will generate identical steps leading to the minimum.
It is also shown that, for a quadratic function, the Davidon algorithm has
a simplified version which searches in the same directions. However, the unique
advantage of the original scheme, that it yields the curvature of the function at the
minimum, is sacrificed for simplicity.
,T
Although these results all apply to the case of a quadratic function, a
comparative study of the same algorithms for a general function can be found
in a companion paper.
* The research reported in this paper was carried out under Contract NAS 9-40369
NASA Manned Spacecraft Center, Houston, Texas.
** Mathematical Analyst
$	 1
INTRODUCTION
Consider the problem of finding the minimum of an unconstrained function,
f , where f has continuous partial derivatives of at least second order. Then
f can be expanded about the minimum point x = h in the Taylor series:
f(x) = f(h) + 2 (x-h)T A(x-h) + R
where	 n-vector of variables and A is the positive definite Hessian
matrix.	 , remainder R estimates the error between f and its quadratic
approximation. Since R becomes negligible when x is sufficiently close to
h, the general function is approximated by a quadratic in a neighborhood of
the mint/ -) 1xm; hence, algorithms converging rapidly for quadratics are of
interest,
Two methods for locating the minimum of a general function are examined
in this paper for their terminal phase properties, that is, R is assumed to be
zero and f is quadratic. A more complete picture can be found in a companion
paper (Ref. 1) where a comparison of the analytic and numeric properties of the
same methods is presented for the non-quadratic case. Since f is assumed
quadratic, its gracliew vector g(x) equals A(x - h) and is represented ,at any
point xi by gi.
Both minimizing procedures rely upon the method of successive linear
searches. This means h is located as the limit of a sequence o , x 1 ' xz
where 
o
 is an initial approximation of h , and xi+1 is the minimum of the
function along the line passing through x i in some specified direction pi . The
step A x i from xi to xi+1 equals - k  pi where k  is a scalar step parameter
df(xi - kpi)
satisfying	 d k	 = 0 or equivalently
2
I	 ;
f
f
T= 0
	
for all i(Pi P g3+1 ^
	 pi gi+1
It remains to examine algorithms for generating pi.
A frequent choice for pi is g  , referred to as the method of steepest
descent. Although this method is stable, that is, a decrease in f is assured
after each step, the minimum is approached asymptotically, resulting in very
slow convergence.
It is shown in Reference 2 that if the direction vectors have the property
of A-con jugacy, that is, for a quadratic function:
pi A p j = 0	 1# j
then
Pi gj	 0	 1 < j
which implies convergence in n steps (or fewer should any k1 = 0, i < n).
Two methods generating a sequence of A-conjugate directions for a quad-
ratic function are the Davidon variable metric algorithm (Ref. 3) as reformulated
by Fletcher and Powell (Ref. 4) and the conjugate gradient algorithm of Hestenes
and Stiefel (Ref. 5). The attractive property of rapid convergence, combined
with stability and simplicity (only first order partial derivatives are required),
make these two of the most powerful and practical methods known.
(1)
(2)
(3)
3
By combining (1) with the symmetry of Hi , equation (7) reduces to
I	 I
THE ALGORITHMS REVIEWED
The direction vectors of the Davidon scheme are given by
Pi
	
Hi gi
	0:!r- 1 !5 n
where 
o 
is any positive definite, symmetric matrix and Hip 1 is defined
by the recursion formula:
Hi+1 Hi + Yi z 
where
Ax Axi 
i Ax  dg
	
i	 i
and
Z
Hi O g` A gi Hi
Z.
1	 gi Hi A gi
For a quadratic function, the change in the gradient A gi
 = gi+1 gi	 A 0 xi'
It is shown in Reference 4 that H i is a positive definite symmetric matrix,
from which the stability property is derivaq. A-conjugacy of the direction
vectors is established together with the property that
n-1
A l =Hn
=I Yi
i=0
(4)
(5)
(6)
r
(7)
(6)
$	 !
_	
Hi 0 gi 0 gi Hii^ g T^ Hg +6THi+, i i+1	 i 191
and the following relationships hold
Yi-1 gi = 0
and
T
Z 6 =	
Hi
-1 A gi-1(gi Hi-1 gi)
	
1-1 i	
gi Hi-1 gi + gi-1 Hi-1 gi-1
implying
T
	H 	 - Hi-1 Agi-1(gi Hi-1 gi)pi	 i-1 
i gT H 6+ g T H gi i-1 i
	
i-1 i-1 i-1
o 
is chosen as the identity matrix, causing the first step to be in the direction
of steepest descent.
The direction vectors of the conjugate gradient algorithm, represented
by r  to distinguish them from those of the Davidon meth^^4, are specified by:
(9)
(10)
o go
and
I giI 2
ri	
ri-1 I g 1 2 + gii-1
where Ig.l 2 = gT g,.
lsisn
5
(11)
#	 {
The property of A-conjugacy of the vectors r i is inherent in their deri-
vation via the Gram-Schmidt orthogonalization procedure, from which the fol-
lowing are immediate for a quadratic function
Tgi g 1
	
= 0
	 1#j	 19j5'n
and
g3 ri = gil 2	 i 2 j
Stability is proved in Reference 6.
EIGENVALUES OF THE DAVIDON H MATRIX
Since each Hi matrix: of the Davidon algorithm is positive definite and
symmetric, its eigenvalues are positive real numbers. In particular, it shall
be shown that at least (n-i-1) of these are unity when the function is quadratic.
Lemma 1;gT Hi-1 gi 0	 for i < j s n
From A-conjugacy, we have that
T	 0	 i<jsnSj Pi
so
T	 T
T H	 - gl Hi-1 gi-1( gi Si-1 gi)	 0	 b 10gj i-1 gi	
iT H g 
+ B
	 iT H gi
	
y
gi	 -1 i
	
i-1 -1 -1
(12)
6
But
T	 T
gj
 Hi-1 ^'gi-1 i gj Hi-1 gi by (3)
implying
.	 T
g H - gT	 (^	 i	 i_1 i
gj Hi-1 8i L 1 -^ T	 T	 ]= 0
gi Hi-1 gi + gi-1 Hi-1 gi-1
For the factor on the right to be zero, gT Hi-1 gi-1 must be zero, which is
impossible since gi-1 is assumed non-zero and Hi 1 is positive definite.
So 91 Hi-1 gi 0 and the lemma is proved.
Theorem 1: Hi fi = 9 	 i < j s n
Hi g
i 
= Hi-1 gi + Yi-1 gj Z1-1 gj 	 by (5)
where
Yi-1 gj = 0
	
by (3) and (5)
and
Z=	 Hi-1 A gi-1	 T H
	 - T H	 bi-1 gj	 Og T H Og	
( gi i-1 gj gi-1 i-1 gj)	 y ( 7)
i-1 i-1	 i-1
But
gi Hi-1 gj 0
	
by Lemma 1
and
7
}
tg 1 Hi-1 gj 	 A	 by (3)
implying
Hi gj = Hi-1 gj
By repeated application
Hi gj Hi-1 gj =	 = o gj
This demonstrates that the gradients gj are eigenvectors with unit eigenvalue
of the matrix Hi , i < j s n.
An immediate consequence is the mutual orthogonality of the gradient
vectors, for if
gi gj =gi Hi gh 	 i<j5 n
then
T	 "'
gi gj = pi' gj = 0	 by (3)
and, by symmetry,
gig =0
j
Since r^ut:al orthogonality of non-zero vectors implies linear independence,
it is confirmed that unity is an eigenvalue of Hi of multiplicity n-i-1.
A further consequence of the theorem is that the algorithm for the
direction vectors (10) for a quadratic function is reduced to the recursion
formulas
(13)
8
(g g^-i + Ig,l I)pi-^ 	 0<iyn	 (14)
Igil	 1-1 i-1
EQUIVALENCE OF THE CONJUGATE GRADIENT AND
DAVIDON METHODS FOR QUADRATIC FUNCTIONS
By an inductive argument, it shall be shown that p i as given by (14)
and r  as given by ( 11) are scalar multiples of each other. This implies that,
at each step, the Davidon and conjugate gradient schemes search along the same
straight line, Assuming the one-dimensional search for the minimum of f
along this line is free from numerical error, both methods will generate iden-
tical s3quences leading to the minimum of the quadratic function f,
Theorem 2: pi = ai r1	 0 S i< n
where cii is a positive, non-zero scalar
Since
r =go = p	 and	 a = 10	 0	 0
both methods have identical values for x1 and g10
and
( ggT+ Ig I 2 I )P	 g.. 1 1 2+ 1 g 12
	
__ __1 0	 1	 0 	 ^ go 	 1 0
p1	
Ig 1 2 + gT p	 Ig 1 2 +i g 12
	
],	 o 0	 1	 0
ig1I2
	
go Ig1 1 2 + g1 1 gA2r .^ - 0 -- 2 + g1
 =	 2
I g01	 1 gol
9
implying
Pi - a1 r1
$	 with
I gol
2
a1 =	
2 + 
l 2 > 0I gil	 l^
Assume pi = a3 rj with a
i 
> 0 for all J s 1 < n and therefore both
algorithms generate identical values for x k 9 gk	 k = 0 f, 1, - - -, i, i+1
T	 2
_ gi+1 gi pi + Igi+li pi
pi +1 -
	
2 + T
	
by ^ 14)
	
l gi+l)	 gi pi
2
gi+1 gi a  r  + I gi+i l a  r 
-	 by inductive hypothesis
I g I 2 + gT a
i	
r
	
+1	 i i i
 Igi+1
g . 1
2 
+a.Ig. 1
2 lgil2+ Igi+lI ri] by (12)
+1	 i i
of I g•i 2 	 1 gi+11
2
Egi+1 +	 2
ig.,I2+a•Ig.IG 	 Ig .I	 1i+^	 i i	 i
ai+1 ri+1	 with	 ai+l> 0
and the theorem is proved,
10
O`METRIC INTERPRETATION OF THE DAVIDON H MATRIX
FOR QUADRATICS FUNCTIONS
Because of the relationship between p i
 and ri , the space W spanned
by pi , - - -, pn_ is also spanned by ri , --- , n-1
.
 It is shown in Reference
6 that
i-1
ri = c  gi + I c.J A rj	 i < nj=0
where c 	 j = 0, - - -,1 are scalars. Since the vectors A r  ( j = 0 9- - -Vi-1)
span Wl, the orthogonal complement of W, r  is a scalar multiple of the
projection of gi
 on W.
By noting that
A p. - k A Ax.	 k A gJJ	 j
the inductive proof of Theorem 3 demonstrates that p i is the projection of gi
in W.
i-1
Theorem 3: pi = gi - I s,J Agj	i < nj=0
where s  are scalars
Since po = gp , the induction is valid for i = 0. Assume
x
i-2
pi-1	 gi-1 -	 c• OgJ	 jj=0
i < n
11
Iwhere c  are scalars	 J = 0, - -, 1-2,
Combining the inductive hypothesis with (13) gives
`	
2gi-1 pi -1 = (1 +c t-2 ) Igi-11
Substituting (16) and the inductive hypothesis into (14) gives that
2	 2	 2 i-2gi(1 + 
c1-2 ) Igi- 1I + Igi) 9i-1 - I gil I of L1gj
_
J=O
pi	 2 + 1 + c
	
2
	
i gi I 	 (	 1-2) I gi-11
Defining
Igii2
Si -1	 I g 1 2 +( 1+c	 )Ig	 I2i	 1-2	 i-1
and
s 
	
si-1 c 
	 j= 0 1 - -, i-2
gives
1-2
pi gi - si-1 Agi-1 - 1 S  Agjj^o
and the theorem is proved.
Since pi = Hi gi , it follows that the transformation represented by
the Hi
 matrix projects the gradient gi
 into W. This projected gradient be-
comes the next direction of search for the minimum of the quadratic function
in the subspace W.
(16)
iQi+1	 I - E Z
j=0 J
i < n	 (21)
I	 i
A MODIFIED VERSION OF THE DAVIDON METHOD
Noting the disappearance of the matrix Yi in equation (10), it is tempting
to redefine the Davidon direction vector p i as
	
Pi	 Qi gi	 i=0,---,n
where Qo = o and
Qi+1 Qi - z 
Qi 0 gi d gi QiZ = i —i
	
i	 Ag 
Qi A gi
and investigate the relationship between pi and pf . It is quickly seen that
Qi is symmetric and positive semi-definite, but it cannot be immediately
assumed that the direction vectors' are A-conjugate.
With o = Qb = 1, formula (5) can be written
i
Hi+1 I + Y - Zj	 j
j=0	 j=0
i < n (20)
and (18) can be written
Showing that Z = Z^ (j = O, - - -, n-1) is then equivalent to proving that p, p.'j	 1	 1
(i = 0 9 - - -, n).
13
l iqlplqjpo^ : 1 pill vol!Plow!
1f	 ^
Theorem 4: The Davidon method and its modified version generate identical
direction vectors for a quadratic function
By definition
^	 ego &gb
o - 0 0 TA
0 0
Assuming Z  = Zj (j = 0 9 - - -, 1-1) implies that
H  gk = Qk gk	 k = 0, - - -, i
and
i-1
Hi = Qi +1
j=0
The inductive hypothesis (22) implies that g+1 pi-1- 0 and g T1 pi-1 # 0 by
properties ofpi-1. Starting with gi+1 pi = 0 by (1) and substituting these
properties in the argument of Lemma 1, it is seen that g +1 Qi-1 g  - 0. The
symmetry of Qi-1 , together with the above relationships and the logic of
Theorem 1, will give the desired result that
Qi gi+1 = gi+1
Substituting (24) into (19) gives
(g	 i- Q g.) ogT QiZ. 	 +1 	 i	 i	 ii	 AgT(gi+1- Qi 9i )
(22)
(23)
(24)
By (7) and Theorem 1
( gi+1 - Hi gi ) A gi HiZ. =
i	 O gi (gi+1- Hi gi )
By (23)
T	 T	 i-1 (ogi ©x )OxT
A gi Hi = 0 gi Qi +
	
T 3	 3
J=0	 Ax j  A g3
which by (3) reduces to
0 gT 	 ► iHi	 g  Qi
Using (25) and (22) it is shown that
Z	 Z.1	 1
and therefore
I
Pi	 pi
and
i-1
Hi = Qi +	 Y.J
J=0
for all values of i s n.
Although both algorithms generate the same steps to the minimum of
the quadratic function, only the original Davidon H matrix yields the curvature
of the function at that minimum. In particular, by comparing (26) and (8), it
15
(25)
(26)
is seen that H  = A 1 while Q  = 0. The implication for a non-quadratic function
is that Q  should be replaced b the estimate of A 1	 - Yi and the	 y	 given by Ti
i=0
same process repeated every n iterations.
CONCLUDING REMARKS
For a quadratic function, it has been shown that the Davidon method, its
modified version, and the conjugate gradient method theoretically generate iden-
tical steps leading to the minimum. Reference 1 contains a comparison of the
numerical efficiencies of the schemes when programmed for the IBM 7094 com-
puter, using identical routines for the one-dimensional search (Ref. 8) ar.,d the
closed form evaluation of a quadratic function and its gradient. By using the
Davidon method and its modified version, almost identical solutions were ob-
tained which were more accurate than those obtained using three theoretically
equivalent representations of the conjugate gradient scheme.
Also contained in Reference 1 is a comparative study of the same five
algorithms applied to a non-quadratic function. The Davidon method, closely
followed by its modified version, clearly exhibited the best convergence in
ex nples with small numerical error, that is, with all computations done in
double precision arithmetic (sixteen decimal digits) and with closed form evalu-
ation of the function and its gradient. To simulate the effects of numerical error,
the computations were repeated in single precision (eight decimal digits). The
deterioration exhibited by all the algorithms was especially severe for the Da-
vidon scheme on account of the appearance of gradient differences in the metric.
Less seriously affected was the modified Davidon algorithm, probably because
of the systematic discarding of this error accumulation every n cycles, How-
16
1ever, both schemes were ultimately susceptible to loss of the descent property.
The simplest version of the conjugate gradient algorithm proved the least sus-
ceptible to error propagation influences.
i
ACKNOWLEDGMENT
The author is indebted to Dr. Henry J. Kelley whose suggestions and
encouragement provided the inspiration for this paper.
YREFERENCES
1. Kelley, H. J. and Myers, G. E. ; "'Conjugate Direction Methods for
Parameter Optimization," presented at the 18th Congress of the Inter-
-national Astronautical Federation, Belgrade, Yugoslavia, September
1967. To appear in Astronautica Acta.
2. Fletcher, R. and Reeves, C. M.; "Function Minimization by Conjugate
Gradients," . Computer Journal, July 1964.
3. Davidon, W. C. ; "Variable Metric Method for Minimization," Argonne
National Laboratory Report ANL-5990 Rev., November 1959.
4. Fletcher, R. and Powell, M. J. D. ; "A Rapidly Convergent Descent
Method for Minimization," Computer Journal, July 1963.
5. Hestenes, M. R. and Stiefel, E. ; "Method of Conjugate Gradients for
Solving Linear Systems," Report 1659, National Bureau of Standards,
1952.
6. Beckman, F. S. ; "The Solution of Linear Equations by the Conjugate
Gradient Method," in Mathematical Methods for Digital Computers ,
A. Ralston and H.S. Wilf, Editors, John Wiley and Sons, New York,
N.Y. , 1960.
7. Hestenes, M. R. ; "The Conjugate-Gradient Method for Solving Linear
Systems," in Proceedings of Symposia in Applied Mathematics, Vol. 60
McGraw-Hill Book Co., New York, N. Y. , 1966.
18
IJohnson, I. L. and Myers, G. E. ; "One-Dimensional Minimization Using
Search by Golden Section and Cubie, "Fat Methods," NASA Manned Space-
craft Center Internal Note No. 67-FM-172, November 13, 1967.
0	 f
19
M
.,	 , .


Properties of the conjugate gradient and Davidon methods, part 5 Final report

Abstract

Similar works

Full text

Available Versions

NASA Technical Reports Server