Abstract. The family of algorithms introduced by Broyden in 1965 for 
solving systems of nonlinear equations has been used quite effectively on 
a variety of problems. In 1979, Gay proved the then surprising result 
that the algorithms terminate in at most 2n steps on linear problems with 
n variables. His very clever proof gives no insight into properties of 
the intermediate iterates, however. In this work we show that Broyden's 
methods are projection methods, forcing the residuals to lie in a nested 
set of subspaces of decreasing dimension.
(Also cross-referenced as UMIACS-TR-93-23

O'Leary, Dianne P.

Digital Repository at the University of Maryland

WHY BROYDEN'S NONSYMMETRIC METHOD TERMINATESON LINEAR EQUATIONSDIANNE P. O'LEARYAbstract. The family of algorithms introduced by Broyden in 1965 for solving systems ofnonlinear equations has been used quite eectively on a variety of problems. In 1979, Gay provedthe then surprising result that the algorithms terminate in at most 2n steps on linear problems with nvariables. His very clever proof gives no insight into properties of the intermediate iterates, however.In this work we show that Broyden's methods are projection methods, forcing the residuals to liein a nested set of subspaces of decreasing dimension. The results apply to linear systems as well aslinear least squares problems.Key words. Broyden's method, nonlinear equations, quasi-Newton methods.AMS(MOS) subject classications. 65H10,65F10August 31, 19941. Introduction. In 1965, Broyden introduced a method for solving systems ofnonlinear equations g(x) = 0, where g : Rn ! Rn is dierentiable [2]. He named ita quasi-Newton method. Methods in this class mimic the Newton iterationxk+1 = xk   frg(xk)g 1g(xk); k = 0; 1; :::;by substituting an approximation Hk for frg(xk)g 1, the inverse of the Jacobianmatrix. This matrix approximation is built up step by step, and the correction to Hkis fashioned so that the secant conditionHk+1yk = sk;is satised, where yk = g(xk+1)   g(xk)  gk+1   gk;and sk = xk+1   xk:This condition is satised if Hk+1 = frg(xk+1)g 1 and g is linear, and this is themotivation for choosing the correction to Hk so that Hk+1 satises this condition butso that H 1k+1 behaves as H 1k in all directions orthogonal to sk. This reasoning ledBroyden to the formula Hk+1 = Hk + (sk  Hkyk)vTk ;where vk is chosen so that vTk yk = 1. Specic choices of vk lead to dierent algorithmsin Broyden's family.Although certain members of the family proved to be very eective algorithms(specically, the so called Broyden \good" method that takes vk in the same directionas sk), the algorithms were little understood, to the extent that Broyden himself in Department of Computer Science and Institute for Advanced Computer Studies, University ofMaryland, College Park, MD 20742. This work was supported under grant NSF CCR 9115568.11972 said, \The algorithm does not enjoy the property of quadratic termination evenwhen used for function minimization with exact line searches." [3, p.97].This misconception, common to the entire optimization community, was disprovedin 1979 by David Gay in a surprising and clever construction [4]. He showed that ifg(x) = Ax  bwhere A 2 Rnn is nonsingular, then the following result holds.Lemma 1.1. (Gay) If gk and yk 1 are linearly independent, then for 1  j b(k+1)=2c, the vectors f(AHk 2j+1)igk 2j+1g are linearly independent for 0  i  j.This result implies that for some k  2n, gk 1 and yk 2 must be linearly depen-dent, and Gay showed that in this case gk = 0 and termination occurs.Gay's construction leads to a proof of a 2n-step Q-quadratic rate of convergencefor Broyden's good method. It yields little insight, though, into how the intermediateiterates in Broyden's method are behaving in the case of linear systems, and thepurpose of this work is to develop that understanding.2. The Character of the Broyden Iterates. First we establish some usefulrelations. The change in the x vector is given bysk =  Hkgk;and the change in the residual g isyk = Ask =  AHkgk;so we can express the new residual asgk+1 = gk + yk = (I  AHk)gk:For convenience, we denote the matrix in this expression asFk = I  AHk;and denote the product of such factors asPk = FkFk 1:::F0 = FkPk 1:Then a simple induction-style argument gives us a useful formula for gk+1 and thusfor yk and sk.Lemma 2.1. gk+1 = Pkg0;yk =  AHkPk 1g0;sk =  HkPk 1g0:Thus, the character of the product matrices Pk determines the behavior of theresiduals gk in the course of the iteration. The key to this behavior is the nature ofthe left null vectors of Pk, the vectors z for which zTPk = 0. We will prove in the nextsection that these vectors have a very special form. In particular, the factor matrices2Fk are quite defective, having a zero eigenvalue with a Jordan block of size dk=2e.The linearly independent left principal vectors of Fk, vectors zi satisfyingzT1 Fk = 0;zT3 Fk = zT1 ;zTi Fk = zTi 2;do not depend on k, and in fact are left eigenvectors of Pk corresponding to a zeroeigenvalue. Thus the vector gk+1 is orthogonal to the expanding subspace spannedby the vectors fzig (i odd and i  k) and, after at most 2n steps, is forced to be zero.3. The Behavior of the Broyden Iterates. David Gay proved an importantfact about the rank of the factor matrices Fk, and the following result is essentiallyhis. Lemma 3.1. (Gay) For k  1, if yk 6= 0, vTk yk 1 6= 0, and rank(Fk) = n 1, thenrank(Fk+1) = n  1, and yk spans the null space of Fk+1.Proof. For k  0, Fk+1 = I  AHk+1= I  AHk   (Ask  AHkyk)vTk= I  AHk   (yk   AHkyk)vTk= (I   AHk)(I   ykvTk )= Fk(I   ykvTk ) :Since vTk yk = 1, we see that Fk+1yk = 0. Similarly, yk 1 spans the null spaceof Fk (k  1). Any other right null vector y of Fk+1 must (after scaling) satisfy(I   ykvTk )y = yk 1. But vTk spans the null space of (I   ykvTk )T , and because, byassumption, vTk yk 1 6= 0, yk 1 is not in the range of (I   ykvTk ), and thus yk spansthe null space of Fk+1.This lemma leads to the important observation that the sequence of matrices Hkdoes not terminate with the inverse of the matrix A, at least in the usual case inwhich all vTk yk 1 6= 0. In fact, each matrix Hk and A 1 agree only on a subspace ofdimension 1.We will make several assumptions on the iteration:1. vTj yj 1 6= 0, j = 1; 2; :::; k, and yk 6= 0.2. v0 is in the range of FT0 .3. k  2n  1 is odd. (This is for notational convenience.)We can now show that the matrix Fk is defective and exhibit the left null vectorsof Pk (which are the same as the left null vectors of Pk+1).Lemma 3.2. Dene the sequence of vectorszT1 F1 = 0; zTi Fi = zTi 2 i = 3; 5; :::; k:These vectors exist and are linearly independent, and zTi is a left null vector of Pj forj = i; i+ 1; :::; k. Further, zTi gj = 0, j = i+ 1; i+ 2; :::; k.Proof. Dene z1 by zT1 F0 = vT0 . This nonzero vector exists by Assumption 2, andsince Fj = F0(1  y0vT0 ):::(1  yj 1vTj 1);3it is easy to see that, for j = 1; 2; :::; k, zT1 Fj = 0, and therefore zT1 Pj = 0 as well. Inorder to continue the construction, we need an orthogonality result: for j = 2; 3; :::; k,zT1 yj = zT1 gj+1   zT1 gj = zT1 Pj g0   zT1 Pj 1 g0 = 0:For the \induction step", assume that linearly independent vectors z1; :::; zi 2have been constructed for i  k, that each of these vectors satises zTmFj = zTm 2(zT1 Fj = 0) and zTmPj = 0 for j = m; :::; k, and that zTi 2yj = 0 for j = i   1; :::; k.(Linear independence follows from the fact that they are principal vectors for Fi 2.)Let zTi Fi = zTi 2. Note that zi exists since zTi 2yi 1 = 0, and yi 1 spans the null spaceof Fi. We have that, for j = i; :::; k, zTi Fj = zTi 2, and therefore zTi Pj = zTi 2Pj 1 = 0as well. Then for j = i + 1; :::; k, we havezTi yj = zTi gj+1   zTi gj = zTi Pj g0   zTi Pj 1 g0 = 0:In the course of this proof, we have established the following result.Theorem 3.3. Under the above three assumptions, and if A is nonsingular, thenafter k+1 steps of Broyden's method, the residual g(xk+1) is orthogonal to the linearlyindependent vectors z1; z3; :::; zk, and thus the algorithm must terminate with theexact solution vector after at most 2n iterations.4. Overdetermined or Rank Decient Linear Systems. Gerber and Luk[5] gave a nice generalization of Gay's results to overdetermined or rank decientlinear systems, and the results in this paper can be generalized this way as well. Thematrices Fk have dimensions m m, where m  n is the number of equations. Thetwo lemmas and their proofs are unchanged, but the theorem has a slightly dierentstatement. Let R denote the range of a matrix, and N denote the null space.Theorem 4.1. Under the above three assumptions, and if A 2 Rmn has rankp, and if N(Hk) = N(AT ), then after k + 1 steps of Broyden's method, the resid-ual g(xk+1) is orthogonal to the linearly independent vectors z1; z3; :::; zk, which arecontained in the range of A, and thus the algorithm must terminate with the leastsquares solution vector after at most 2p iterations.Proof. The fact that zi is in the range of A is established by induction.Gerber and Luk give sucient conditions guaranteeing that N(Hk) =N(AT ):1. R(H0) = R(AT ),2. N(H0) = N(AT ),3a. vk = HTk uk for some vector uk, with uTk sk 6= 0.Condition 3a is satised by Broyden's \good" method (vk parallel to sk) but not the\bad" method (vk parallel to yk). It is easy to show by induction that the conditionN(Hk) = N(AT ) also holds under the following assumption, valid for the \bad"method: 3b. vk = Auk for some vector uk, with uTk sk 6= 0.First, assume that z 2N(Hk) and that N(Hk) = N(AT ). ThenHk+1z = Hkz + (sk  Hkyk)vTk z;and this is zero if and only if vTk z = 0, which is assured if either vTk = uTkHk or vTk =uTkAT . Thus, N(Hk)  N(Hk+1). Now, by writing vk as HTk uk, valid under either3a or 3b, we can show [5] that Hk+1 = (I  Hkgk+1uTk )Hk, and that the determinantof the rst factor is uTk sk, nonzero by condition 3. Thus N(Hk+1) = N(Hk).45. Concluding Notes. Since the vectors vk are arbitrary except for a normal-ization, it is easy to ensure that Assumptions 1 and 2 of x3 are are satised. But ifnot, termination actually occurs earlier: if Fk has rank n   1 and vTk yk 1 = 0, thenFk+1 has rank n 2 with right null vectors yk and yk 1. Similarly,Pk+1 has one extrazero eigenvalue.The conclusions in this work apply to the Broyden family of methods, whetherthey are implemented by updating an approximation to the Hessian, its inverse, or itsfactors. The inverse approximation was only used for notational convenience; otherimplementations are preferred in numerical computation.The conclusions depend critically on the assumption that the step length param-eter is 1 (i.e., if sk =  Hkgk, then  = 1).As noted by a referee, the conclusions depend on only two properties of Broyden'smethod, consequences of Lemma 2.1:Fk+1 = Fk(I   (Fk   I)gkvTk );gk+1 = Fkgk;where g0 and F0 are given, and the vk are (almost) arbitrary. Thus the results can beapplied to a broader class of methods, in the same spirit as the work by Boggs andTolle [1].6. Acknowledgement. Thanks to Gene H. Golub for providing the Gerber andLuk reference, and to David Gay and C. G. Broyden for careful reading of a draft ofthe manuscript. REFERENCES[1] P. T. Boggs and J. W. Tolle, Convergence properties of a class of rank-two updates, SIAMJ. on Optimization, 4 (1994), pp. 262{287.[2] C. G. Broyden, A class of methods for solving nonlinear simultaneous equations, Mathematicsof Computation, 19 (1965), pp. 577{593.[3] ,Quasi-Newton methods, in Numerical Methods for UnconstrainedOptimization,W. Mur-ray, ed., Academic Press, New York, 1972, pp. 87{106.[4] D. M. Gay, Some convergence properties of Broyden's method, SIAM J. on Numerical Analysis,16 (1979), pp. 623{630.[5] R. R. Gerber and F. T. Luk, A generalized Broyden's method for solving simultaneous linearequations, SIAM J. on Numerical Analysis, 18 (1981), pp. 882{890.5

Why Broyden's Nonsymmetric Method Terminates on linear equations

https://drum.lib.umd.edu/bitstream/1903/583/4/CS-TR-3045.pdf

Why Broyden's Nonsymmetric Method Terminates on linear equations

Abstract

Similar works

Full text

Available Versions

Digital Repository at the University of Maryland