Frey, T.

Turi, J.

English

Periodica Polytechnica (Budapest University of Technology and Economics)

ON THE GENERALIZATION OF A QUASI-NEWTONIAN METHOD By , T. FREyl and J. TUR! Department of Mathematics of the Faculty of Electrical Engineering Received February 21. 1979 Presented by Prof. Dr. l. FD;YO 1. Introduction In 1972 Bass has suggested [lJ a variable metric quasi-Newtonian method for minimizing convex functions, which does not need any line-search optimization step. However, the convergency properties of the above-mentioned method are not yet known. A suitable variable metric quasi-Newtonian method is planned to be extended for minimizing convex functionals defined over an infinite dimensional, but separable Hilbert space. The most suitable method for this aim seems to be a new family of variable metric quasi-Newtonian methods, which we have developed from the ideas of Bass. Let us point out, that there is a though unimportant - mistake in the paper by Bass [lJ but for our version it is vital to be improved. Let us outline here first the essence of Bass method, with some important supplements. The third chapter will give a suggestion for a new family of quasi-Newtonian methods, formulated so as to include both the finite and the infinite dimensional case. The convergency properties of the family will be presented in the last chapter. The proofs will only be sketchy. 2. Conjugate directions and inversion of linear operators In what follows, A and B denote symmetric (Hermitian) positive definite operators, mapping some separable Hilbert space onto itself. {d i } and {vJ are two linearly independent and complete sets, assisting the Gram-Schmidt method to build the conjugate directions of A and B- 1 respectively, {ak } and {Cl}' Now, if {ak} is a complete set of A-conjugate directions, i.e. if (1) 48 T. FMI' and J. TURl {Cl} is a complete set of B-i-conjugate directions, i.e. if (2) is valid for all i and j, then also (3) and (4) In fact, mUltiplying (3) by Aa i, (1) leads to: and in the same way, multiplying (4) by B-icj , we get cj for all j. Now, relationships (3) and (4) permit to recursively get the sum (3), starting from the operator 0, provided each of the ak lie in the subs pace spanned by d1 ) dl • ••• , dk, and have the form Gk = dk + I: Cf.ikd. i<k (5) In a more or less similar way, starting from the operator B and ending \'1ith the operator ° yields (4), provided each of the c lie in the subspace spanned by Bu l ) ... , BCI' and have the form (This is exactly the mistake by Bass, namely that c is iying in the subspace Vi' Vl' •.. , VI. In fact, however, Bass needs only (3) in his method.) Theorem 1. The recursions (7) and GENERALIZATION OF A QUASI-.'iEWTONJAN METHOD c.c! B.--'-' , V!c. , , satisfy the above mentioned conditions. 49 (8) Sketch of the proofby induction. First, ai + 1 is lying in the subs pace spanned by d 1 , d2 , ..• , di' di+ 1, and has the form (5), if this fact holds for j = 1,2, ... , i, because then Ai is a projections operator into the subspace spanned by a1, aZ, ... , ai • Second, Qi+ 1 is an A-conjugate direction with respect to a1 , a2 , ••• , ai' if this fact holds for j = 1, 2, ... , t, namely from (7) =aJAdi + 1 Third, c;+ 1 is lying in the subspace spanned by Bv 1, Bv2 , •.. , Bv;+ l' and has the form (6), if this fact holds for j = 1, 2, ... , t, because, by the inductive definition in (8), B;+ 1 has the form (9) where the projections operator Pi projects onto the subspace spanned by Cl' C2 , •.. , Cb hence, according to our hypothesis, into the subspace of Bv1 , Bv2 , .•• , Bv;, and therefore (10) has the form (6). Fourth, c;+ 1 is B- 1 -conjugate with respect to C1,C2' ... ,Cl' if this fact holds for j = 1, 2, ... , i, because from (8) and (9): 50 =C~V'''''l J I. =C~V'''''l J I. T. FREYand J. TURf 3. Suggestion on the new family q. e. d. Calculating the change of the gradient for a given discharge is known to yield some information on the local second derivative operator, at least on its mapping with respect to the direction considered. All of the single-and double-rank variable metric quasi-Newtonian methods apply just the above information for updating the approximate local Newtonian operator. Except Bass's method, however, all the others are using only once the above mentioned information in each step via the optimizer namely that provided automatically by the applied method. According to [lJ, information may be collected in several directions on the second derivative before updating it, independent of whether information is collected by stopping at a step and sweeping over the space or proceeding continuously towards the optimizer. Bass suggest to use (3) and (4) for updating both the local Newtonian operator, and the local second derivative operator. Namely the projections operators, derived from (3), have to be added to the new Newtonian operator, whereas the projections operators, derived from (4), have to be substracted from the former one. In the Bass version however the second half of this procedure is unimportant, because in each cycle as many information is collected in independent directions, as the basic space has dimensions, resulting in each cycle in the full sum of (3). This is why the mistake of Bass with respect to the subspace of the c's is unimportant. This suggestion is remarkable, because in other known methods the search directions are not freely chosen and therefore passing to all possible directions at an infinite frequency on the Newtonian operator cannot be taken as granted. Therefore in general there is no guarantee that the approximation of the Newtonian operator will converge to the right GENERAliZATION OF A QUASI·NEWTONIAN METHOD 51 value. Bass's suggestions are felt to lead to the required convergency, but this is still to be demonstrated. Also the most important in Bass's suggestion is to be free to chosen the search direction, while the other aspect-namely to collect in each cycle as many information as there are dimensions before updating the Newtonian operator-is not essential. Collecting however in each cycle less information than the number of dimensions imposes to cleverly update the approximation of the Newtonian operator aptly combining (3) and (4). Thereby it becomes unimportant whether the basic space is finite dimensional, or not. This method can be used to minimize convex functionals with sufficiently smooth second derivative, define~ over a separable Hilbert space, and, the next Chapter will show it to improve the convergency over that by any other method. Our suggestion is the following: in each cycle information is collected on the Newtonian operator in at least two independent directions of which at least one is chosen so that in n steps at least n + 1 independent directions are tested. One of the at least two directions per cycle will be imposed by the method itself, so as for any other variable metric quasi-Newtonian method (namely for this step the information results without extra calculations). The other directions will be rotated, to infinitely often sweep the full space. It means that in the case of infinite dimensions at least two directions are needed per cycle i.e. at least three directions together with that imposed to by the very method. One possible choice of this two independent rotating directions in the infinite dimensional space is by providing the space a basis -maybe an orthonormal one-. In one of the two directions the basis is swept over once systematically, and while this direction takes the basic vectors with subscripts 211 + 1 and 211 + 1 the other takes the basic vectors with subscripts 1 to 211. To update the Newtonian operator, (3) and (4) are systematically used; (3) in the chosen directions, and with the local new Newtonian operator, to be approximated, where A in (3) denotes the unknown local second derivative operator. Also (4) is used, where B means the former approximation of the Newtonian operator to be updated. The most essential idea in our suggestion is how to choose directions Vi in (4). Our proposition is the following: for N directions per cycle, let for (11) and for i>N (12) 4* 52 T. FREY and J. TURl where the coefficients 9ik are chosen so as to assure the orthogonality of Vi to cj (i> k; j = 1, 2, ... , k). Remark that (12) has only a theoretical importance and therefore no coefficients 9ik are effectively needed. To be more precise, Adi = f).gi replaces (3), where di is the ith direction in the cycle considered, and L1gi is the change of first derivative upon changing the independent variable by di• To be sure that the computed directional difference of the first derivative is close enough to the differential of the first derivative, Ildill = (!)(h) must hold, where h denotes an upper bound of the instantaneous distance of the basic point (the temporary approximation by the minimizer) from the minimizer. Our updating formula is the following: (13) where B.\ is generated by (4) from B=B101dl" A family results, with each member defined by N (N ~ 2 in the finite, and N ~ 3 in the infinite dimensional case), and by the choice of the rotating directions in the cycles. 4. Local convergency investigations Let us assume now that the functionalfto be minimized is convex, and its second derivative uniformly satisfies a Lipschitz condition. Therefore for the global convergency investigations the ideas by Broyden, Dennis and More (Chapter 3 in [2J) can be made use of. However, to use these general idea~ the estimates of the norms of [A -1 -(As + Bs)] are needed. Three types of norms will be applied. First the ordinary operator norm, denoted by iI.ll, second, a transformed vector norm, and the corresponding operator norm, defined by a symmetric (Hermitian) positive definite auxiliary operator H, denoted by JI.IIB> and defined for vector v by IlvllH = IIHI/2vll, and for the operator Vby and third, a Frobenius-type norm, defined a symmetric (Hermitian) auxiliary operator S and by an even number k, denoted by 11.1i~·.< and defined the operator V by GENERALIZATION OF A QUASI·NEWTONIAN METHOD 53 where )'; is the ith eigenvalue of SVS. First some estimation are needed for (A k + Bk)Aa i' where A is a symmetric positive definite operator, {d;} is a complete basis of the separable Hilbert space considered, aj and Ak are the sequences generated by (3), B = (A + c:AR) -1 is an approximation of A - 1, good enough (i.e. c: is sufficiently small), itself symmetric and positive definite (i.e. R is symmetric too), VI is the sequence generated by (11) and (12), finally, Bk denotes the sequence of operators, generated by (4), with the directions VI' Theorem 2. ANAa;=a; if N>' =7, AsAa;=O, if N <i; (14) Bsv;=O, if N~i; B.vv;=Bv;, if N<i; (15) (16) are valid. Sketch of the proof: Relationships (14) are trivial consequences of (3), because the projections operators in Ak and in A 1 have the form and the an are A conjugate to am' for n =1= m. The first group in (15) is also a trivial consequence of (4), because the projections operators in B and in Bk have the form (cnc~)/(v~cn)' and the Cn are B-1 conjugate directions, hence if n> j. Ck .J.1 Ck.J.1 Now, from (4), the first operator in Bk is T' " and therefore Bkvj=O Vk + 1Ck + 1 holds for k ~j.The second group in (15) is a trivial consequence of (12), because B and Bk differ only in projections operators of the form (cnc~)/(v~cn) where n;;ik, and from (12) they are all orthogonal to vi' ifj>N. Also (16) is trivial for i;;i N, because for these indices Vi = Aai, and therefore A - 1 V; = ANV; and BNV; = 0, i.e. [A 1 - (AN + BN)] Vi = ° for these subscripts. The only crucial point is the 54 T. FREYand J. TUR] estimation [A -l-(AN+BN)JVi = [A -l-(AN+B)Jvi for i>N, because ANvi=F0 from (12). To this aim will be computed all the quantities of interest in the order of e. Assume: B= [A(l +eR)]-l "'(l-eR)A- 1 (17) To assure the orthogonality of Vi to cj ' coefficients 9ij will be chosen to hold vTBvj = vJBvi=O for i;;£N and}> N, because from Theorem 1, cj is lying in the subspace spanned by Bv 1, Bv2 , .•. , Bvj • Hence aTA(l-eR)A -l[Aaj + L: 9j1Aaa ",0 (18) I:;;; " This equation - by inductive demonstration, of 9j1 = 9(e) yields the estimation (19) From (19) and (3): (20) whereas from (15) (21) therefore (22) GENERALIZATION OF A QUASI-NEWTONIAN METHOD 55 whereas (23) But according to the third relationship (20) ANvj is the projection of -sRa j onto the subspace spanned by vectors an = A - 1 Vn (n = 1, 2, ... , N). (In fact, the projection of a vector w onto this subspace is given - considering the A -conjugateness of the an - in the form w = L wiai, with if w = - sRaj , then (24) and this the same sequence of coefficients as that in (20), q.e.d.) Now (23), (20) and (24) show that (16) is valid, because all the maps of Vj with j> N are diminished by its projection onto a subspace, substituting the operator [A -1 -(AN + B"JJ for [A -1 - Bl Of course this fact does not generally assure that 11 [A -1 (As + B.,,)Jvjll ~ II[A -1 - BJvjll is valid, because there is no information about the signs of aTa j for i~N andj>N, but (17) is valid, because ai is A orthogonal to aj for these subscripts. TIleorem 3 where q(s; N, k) < 1 if s is small enough, (and if the sequence of the eigenvalues of AR tend to 0 fast enough in the infinite dimensional case). Sketch oJ prooJ: (25) is a trivial consequence of (16), because the mapping Qf the complete basis {VI} by [A -1 - (AN + BJJ or by [A - 1 BJ is divided in two A conjugate subspaces, and in the second of them (spanned by aj with j> N) they coincide, whereas in the first the mappings by the first operator vanish, whereas by the second operator not. The proof of (26) needs great many calculations; here the case k = 2, will be insisted on giving only the details of the relationship between and 56 T. FREY and J. TURf The ideas to be applied on the relationship between and are the same, and give a similar inequality, and they together confirm (26). The basic idea is that the norms to be considered are closely related to the trace of the square of the operator - this is well known in the finite dimensional case, but it is easy to extend to the infinite dimensional case, where the trace can be defined by infinite dimensional determinants, as limits of its finite slices. To this aim some estimates in the order of 8 are needed. Using short notations: we get (27) and (28) hence (29) and (30) GENERALIZATION OF A QUASI-NEWTON/AN METHOD With the above we get {IIA -1 -BIIJA;2}2 ={IIA -1-(Al +B1)-I<i alar + +e~ (RalaI +alaIR)II~!A'2}2 = {IIA -1 -(A 1 + Bl)ll~iA,2}2 + (Xl +2tr{JA{[(A- 1-Al) BIJ.[ -e~ialaI.jA+ +e~RalaI.jA+e~alarR.jAJ}~ (Xl (Xl "'{IIA-1 (A B )IIF_}2 (2pi 2 I 2 1 2 = - 1+ 1 . ./A;2 + e (Xt(XITe (XiP1+ +2e2 :i (XlaIRARa l )+2 {o+o+o+tr{vlA (eRA -1 + Si that is the square of the Frobenius-type norm of (A - I - B) exceeds that of [A- I -(A I +B1)J by a term proportional to the square of the norms of the former. 58 T. FREY and J. TURf Using the above yields a similar additive term in all steps which proves our assertion. Results from using the general ideas of Broyden, Dennis and More Theorem 4 If our assertions are valid, then the sequence (AN + BN) tends toward the right value of the local Newtonian operator, hence the sequence for approximating the minimizer is Q-superlinearly convergent. If our space is finite dimensional, than (AN+BN) tends in geometrical order toward the local Newtonian operator. In the Frobenius-type norm also in the infinite dimensional case a geometrical convergence rate toward the local Newton operator, results if our first approximation was in this sense good enough. In the bdlmensional case the rate of convergence is of order 1: where 1: is the positive root of the equation considering stepwise instead of cycle wise convergence-rate. Summary Extension of one suitable variable metric quasi-Newtonian method for minimizing functionals defined over an infinite dimensional but separable Hilbert space is attempted. The most suitable to this aim seems to be a new family of quasi-Newtonian methods, developed from the ideas of Bass. Recalling the essentials of Bass's method with some important supplements, a new family of quasi-Newtonian methods is suggested formulated so as to included both the finite and infmite case. Finally the convergency properties ofthe family are presented. The proofs are only sketchy. References 1. BASS: A rank two algorithm for unconstrained minimization. Math. of Comp. V. 26. 1972. 2. BROYDEN, DENNIS and MORE: On the Local and Superlinear convergence of Quasi-Newton Methods Maths. Applies. V. 12. 1973. Jinos TURI, H-1521 Budapest 

ON THE GENERALIZATION OF A QUASI-NEWTONIAN METHOD

https://pp.bme.hu/ee/article/download/4823/3928

ON THE GENERALIZATION OF A QUASI-NEWTONIAN METHOD

Abstract

Similar works

Full text

Available Versions

Periodica Polytechnica (Budapest University of Technology and Economics)