Lukšan, Ladislav

Vlček, Jan

English

Institute of Mathematics AS CR, v. v. i.

PANM 15Jan Vlček; Ladislav LukšanLimited-memory variable metric methods that use quantities from the preceding iterationIn: Jan Chleboun and Petr Přikryl and Karel Segeth and Jakub Šístek (eds.): Programs and Algorithms of NumericalMathematics, Proceedings of Seminar. Dolní Maxov, June 6-11, 2010. Institute of Mathematics AS CR, Prague,2010. pp. 227--232.Persistent URL: http://dml.cz/dmlcz/702762Terms of use:© Institute of Mathematics AS CR, 2010Institute of Mathematics of the Academy of Sciences of the Czech Republic provides access to digitized documentsstrictly for personal use. Each copy of any part of this document must contain these Terms of use.This paper has been digitized, optimized for electronic delivery and stamped with digitalsignature within the project DML-CZ: The Czech Digital Mathematics Libraryhttp://project.dml.czLIMITED-MEMORY VARIABLE METRIC METHODS THAT USEQUANTITIES FROM THE PRECEDING ITERATION∗Jan Vlček, Ladislav Lukšan1. IntroductionIn this contribution, a new family of globally convergent limited-memory (LM)variable metric (VM) line search methods for unconstrained minimization is pre-sented. Numerical results indicate that the new methods can save computationaltime substantially for certain problems in comparison with the well-known L-BFGSmethod, see [3], [8].VM or quasi-Newton line search methods, see [2], [4], start with an initial pointx0 ∈ RN and generate iterations xk+1 ∈ RN by the process xk+1 = xk+sk, sk = tkdk,k ≥ 0, where dk is the direction vector and tk > 0 is a stepsize.It is assumed that the problem function f : RN → R is differentiable and step-size tk is chosen in such a way thatfk+1 − fk ≤ ε1tkgTk dk, gTk+1dk ≥ ε2gTk dk, (1)k ≥ 0, where 0 < ε1 < 1/2, ε1 < ε2 < 1, fk = f(xk), gk = ∇f(xk) and dk = −Hkgkwith a symmetric positive definite matrix Hk; usually H0 is a multiple of I and Hk+1is obtained from Hk by a rank-two VM update to satisfy the quasi-Newton conditionHk+1yk = sk (see [2], [4]), where yk = gk+1 − gk, k ≥ 0. For i ≥ 0 we denotebi = sTi yi, Vi = I − (1/bi)siyTi(note that sTi yi > 0 for gi ̸= 0 by (1)). To simplify the notation we frequently omitindex k and replace index k + 1 by symbol + and index k − 1 by symbol −.The L-BFGS method (LM variant of the well-known BFGS method, see [3], [8])is based on the following quasi-product form of the BFGS updateH+ = (1/b)ssT + V HV T . (2)The advantage of this form consists in the fact that only the last m̃+1 = min(k+1,m)couples {si, yi}ki=k−m̃, where m ≥ 1 is a given parameter, are stored to compute thedirection vector dk+1 = −Hk+1gk+1 by the Strang recurrences, see [8]. Matrices Hk+1are not computed, only defined by Hk+1 = Hk+1m̃+1, k ≥ 0, whereHk+10 = (bk/|yk|2)I, (3)Hk+1i+1 = (1/bj)sj sTj + VjHk+1i VTj , j = k − m̃+ i, 0 ≤ i ≤ m̃ . (4)∗This work was supported by the Grant Agency of the Czech Republic, project No. 201/09/1957,and the Institutional research plan No. AV0Z10300504.227Note that matrixHk, which satisfies dk = −Hkgk, is different from matrixHk+1m̃ inthe last update (4) in general; among others since matrixHk is created by updating ofmatrix Hk0 = (bk−1/|yk−1|2)I, not Hk+10 = (bk/|yk|2)I. Thus Hk+1m̃ gk ̸=−dk generally.The Strang recurrences cannot be used directly for other updates from the Broy-den class (see [2], [4]) than for the BFGS update (but another efficient approach ispossible, see [6]). Some generalizations of the L-BFGS method are investigated in [9].Here we focus on the approach which uses quantities from the preceding iteration.Note that our methods do not belong to the Broyden class and has some commonfeatures with the multi-step quasi-Newton methods (see e.g. [7]).We describe the new class of VM updates in Section 2 and the correspondingalgorithm in Section 3; global convergence is treated in Section 4 and numericalresults are reported in Section 5. Details and proofs of assertions can be found in [9].2. The new class of methodsThe Broyden class updates except for the BFGS update need calculate vector Hyin every iteration. This drawback can be eliminated by utilization of the quasi-Newton condition Hy− = s−. Although it is not satisfied in LM case, in this waywe can construct efficient methods that use the same number of stored vectors andmatrix by vector multiplications as the L-BFGS method, see Section 3.Theorem 2.1. Let matrix H be symmetric positive definite, Hy− = s−, σ ∈ (−1, 1),s̄ = s − σ√b/b− s−, ȳ = y − σ√b/b− y−, b̄ = s̄Ty ̸= 0 and ϱ̄ = (1 − σ2) b/b̄. Thenupdate HNB+ with parameter σ given byHNB+ = (ϱ̄/ b̄) s̄s̄T + V̄ HV̄ T , V̄ = I − (1/ b̄) s̄ȳT , (5)is positive definite and satisfies the quasi-Newton condition HNB+ y = s (for σ= 0 weobtain the BFGS update and assumption Hy−=s− can be omitted). If σ= sTy−/√bb−then s̄Ty− = 0, b̄ = s̄T ȳ and if also σ ∈ (−1, 1) and b̄ > 0, then (5) represents thegeneralized BFGS update with nonquadratic correction parameter ϱ̄ (see [4]), withvectors s and y replaced by s̄, ȳ. If σ = sT−y/√bb− then sT−ȳ = 0 and ϱ̄ = 1.Our numerical experiments indicate that convergence is significantly deterioratedwhen |σ| → 1 and that all values σ satisfying |σ| ≤ 1/2 with a suitable sign (Theo-rem 2.1 and Lemma 2.1 motivate us to use the sign of sTy−) give very good results.Lemma 2.1. Let Hy− = s− and f be quadratic function f(x) =12(x−x∗)TG(x−x∗),x∗ ∈ RN , with a symmetric positive definite matrix G. If vectors s, s− are linearlyindependent and update HNB+ of matrix H is given by (5) then choice σ = sTy−/√bb−(or equivalently σ = sT−y/√bb−) satisfies b̄ > 0, σ ∈ (−1, 1), ϱ̄ = 1 and HNB+ y− = s−.Note that we need not calculate value sTy−. We use only the sign of sTy−,therefore in view of the following lemma we can utilize the value sT−g, computedduring the line search procedure, in spite of the fact that assumption d = −Hg isnot appropriate to LM updates, see Section 1. In Section 3 we describe a choice ofthe sign of σ in details.228Lemma 2.2. Let H be nonsingular matrix, Hy−=s−.If d=−Hg then sTy−=−tsT−g.Taking into account Theorem 2.1 and Lemma 2.1, we will choose such parameterσ ∈ (−1, 1) that corresponding b̄ is positive and not too small in comparison with bin a sense that b̄ ≡ b(1− σ sT−y/√bb− ) ≥ b(1− λ), λ ∈ (0, 1), which is equivalent toσ sT−y ≤ λ√bb−. The following lemma shows that in case that b̄ < b(1− λ) for someσ ∈ (−1, 1), we can replace this σ by a more appropriate value.Lemma 2.3. Let σ sT−y > λ√bb− for some λ ∈ (0, 1). Then sT−y ̸= 0 and valueσ̂ = λ√bb−/|sT−y| > 0 satisfies ±σ̂ sT−y ≤ λ√bb− (for both signs) and σ̂ < |σ|.3. ImplementationHere we give the procedure based on Section 2. We define matrices Hk+10 andHk+1 = Hk+1m̃+1, m̃ = min(k,m−1), m ≥ 1, k ≥ 0, by relations similar to (3), (4).Instead of matrices Hk, only m̃+1 ≤ m couples of vectors are stored here to computethe direction vector dk+1 = −Hk+1gk+1, using the Strang recurrences, see [8], witha little modification - using transformed nonquadratic correction parameter ϱ̄, see [4].We choose the sign of σ in accordance with the sign of −tsT−g ≈ sTy−, seeLemma 2.2 and Theorem 2.1. Since sTy− = sT−y for f quadratic, see Lemma 2.1,we prefer the sign of sT−y in case that |tsT−g| is too small in comparison with |sT−y|(constant 20 in Step 2 was found empirically). Using Lemma 2.3, we bound |σ| tohave b̄ not too small, compared with b. For simplicity, we omit stopping criteria.Algorithm 3.1Data: The number m of VM updates per iteration, upper bound σ ∈ (0, 1) for|σk|, safeguard parameter λ ∈ (0, 1) and line search parameters ε1 and ε2,0 < ε1 <12, ε1 < ε2 < 1.Step 0: Initiation. Choose the starting point x0 ∈ RN , define direction vector d0 =−g0 and initiate iteration counter k to zero.Step 1: Line search. Compute xk+1 = xk + tkdk, where tk satisfies (1), gk+1 =∇f(xk+1), yk = gk+1 − gk and bk.Step 2: Update preparation. If |sT−y| > 20t|sT−g| set νk = sgn(sT−y), otherwise setνk = −sgn(sT−g). Choose parameter σ̌k ∈ [0, σ] (for k = 0 we choose σ̌k = 0)and set σk = νk σ̌k. If σk sT−y > λ√bb− set σk = λνk√bb−/|sT−y|. UsingTheorem2.1, compute b̄k, s̄k and ϱ̄k and define V̄k.Step 3: Update definition. Set m̃ = min(k,m − 1) and define Hk+10 = (bk/|yk|2) Iand Hk+1 ≡ Hk+1m̃+1 byHk+1i+1 = (ϱ̄j/b̄j)s̄j s̄Tj + V̄jHk+1i V̄Tj , j = k − m̃+ i, 0 ≤ i ≤ m̃ . (6)Step 4: Direction vector. Set k := k + 1 and compute dk = −Hkgk by the modifiedStrang recurrences, using the definition of matrices {Hki }min(k,m)i=0 , and go toStep 1.2294. Global convergenceAssumption 4.1. The objective function f : RN → R is bounded from below anduniformly convex with bounded second-order derivatives (i.e. 0 < G ≤ λ(G(x)) ≤λ(G(x)) ≤ G < ∞, x ∈ RN , where λ(G(x)) and λ(G(x)) are the lowest and thegreatest eigenvalues of the Hessian matrix G(x)).Since our new LMmethods do not belong to the Broyden class, the usual approachmust be modified. The following lemma before the main theorem plays basic role.Lemma 4.1. Let matrix A be symmetric positive definite, ϑ > 0, τ ̸= 0, u ∈ RNand v ∈ RN . Then update A+ given by A+ = τ 2ϑuuT +(I − τ uvT)A(I − τ vuT)is positive definite and satisfiesTr(A+) ≤ τ 2ϑ|u|2 + Tr(A)(1 + |τ |(|u||v|))2, (7)Tr(A−1+ ) ≤ Tr(A−1) + |v|2/ϑ. (8)Theorem 4.1. Let objective function f satisfy Assumption 4.1. Then Algorithm3.1generates a sequence {gk} that either terminates with gk=0 for some k or limk→∞|gk|=0.5. Numerical resultsFirst we demonstrate the influence of parameter σ on the number of evaluationsand computational time, using the collection of sparse and partially separable testproblems from [5] (Test 14, 22 problems) with N = 1000, m = 10, λ = 1/2 and thefinal precision ∥g(x⋆)∥∞ ≤ 10−6.Results are given in Table 1, where ’NFE’ is the total number of function andalso gradient evaluations over all problems, ’Time’ the total computational time inseconds and ϕ is the arithmetic mean of values ’NFE’ and ’Time’ over all σ.σ NFE Time σ NFE Time0.0 22522 8.36 0.3 19854 7.420.033 22185 8.25 0.333 19865 7.360.067 21121 7.80 0.367 20068 7.490.1 20751 7.72 0.4 21359 7.810.133 20940 7.82 0.433 21250 7.820.167 20929 7.77 0.467 20779 7.710.2 20144 7.55 0.5 19754 7.280.233 20579 7.62 0.533 20207 7.390.267 22064 8.08 ϕ 20845 7.72L-BFGS: NFE = 22092 Time = 8.91Tab. 1: Influence of parameter σ for Test 14.230NFV Percentage increase of NFV for σ =Problem N l-bfgs .05 .10 .15 .20 .25 .30 .35 .40 .45 .50BDQRTIC 5000 248 -29 -10 -19 -43 -33 -16 -8 -18 5 -26BROYDN7D 2000 3029 -1 -2 -3 -3 -3 -3 -2 0 2 6CHAINWOO 1000 515 -8 -13 -19 -14 -18 -13 -20 -17 -15 -14CURLY10 1000 5628 4 8 8 5 -5 2 -1 3 -7 -3CURLY20 1000 6852 -6 -7 -6 -9 -9 -7 -10 -9 -7 -10CURLY30 1000 7222 -3 -5 -5 -7 -10 -10 -5 -9 -13 -7DIXMAANE 3000 249 -4 -3 -4 6 -4 -4 -10 2 -11 -10DIXMAANF 3000 189 1 2 14 14 16 14 11 -2 -4 13DIXMAANG 3000 188 11 17 9 5 10 6 13 6 6 -7DIXMAANH 3000 185 7 12 15 10 10 7 -6 5 -4 5DIXMAANI 3000 881 -9 -12 -17 -14 -27 -33 -40 -64 -77 -35DIXMAANJ 3000 317 -3 -3 -4 -5 0 -9 -6 -16 17 20DIXMAANK 3000 270 9 -5 -11 -7 7 4 16 7 37 28DIXMAANL 3000 263 0 -8 -10 -3 -10 -13 -9 8 8 14FLETCBV2 1000 944 28 1 -6 26 35 35 23 54 37 -4FMINSRF2 5625 305 5 1 2 2 2 1 2 8 6 3FMINSURF 5625 460 0 -2 4 13 -6 -18 -4 3 -3 -13GENHUMPS 1000 2223 8 26 14 17 41 19 27 47 52 48GENROSE 1000 2393 -2 -2 0 0 2 3 5 8 10 13MOREBV 5000 116 3 3 -10 -7 -1 -3 -5 -2 0 5MSQRTALS 529 3622 -22 -9 -22 3 -7 -10 -4 -12 -27 -12NCB20 1010 497 3 33 28 7 48 10 -5 25 4 3NCB20B 1000 1792 -5 -23 -5 -5 -8 -9 -9 -12 -9 -6NONCVXU2 1000 3902 -11 -17 -4 4 -2 -13 -9 0 -16 -39NONDQUAR 5000 4244 -17 3 1 3 -1 -11 3 13 -16 -10POWER 500 110 -5 -7 -7 -5 -12 -13 -14 -13 -11 -13QUARTC 5000 236 0 0 0 0 0 0 0 0 0 0SINQUAD 5000 339 5 3 3 -3 10 0 1 11 -3 7SPARSINE 1000 10680 -10 -8 -8 -4 -12 -9 -11 -15 -26 -19SPMSRTLS 4999 224 1 0 -1 0 -5 -2 1 -2 -2 -3VAREIGVL 500 168 -3 -4 -3 -10 -10 -15 -5 -8 -9 -11All problems 58291 -5.6 -3.5 -3.8 -1.2 -4.0 -5.7 -4.2 -2.6 -10.2 -8.1Tab. 2: CUTE - Percentage increase of NFV against L-BFGS.For a better comparison with the L-BFGS method, we performed additional testswith problems from the widely used CUTE collection [1] with various dimensions N ,m = 10, λ = 1/2 and the final precision ∥g(x⋆)∥∞ ≤ 10−6. The percentage increaseof NFV for various values of parameter σ against NFV for the L-BFGS (negativevalues indicate that our results are better than for the L-BFGS) is given in Table 2,where NFV is the number of function and also gradient evaluations. In the last line,the total values over all problems and their percentage increase are given.231Our limited numerical experiments indicate that the suitable choice of param-eter σ can improve efficiency of limited-memory methods, substantially for someproblems.References[1] Bongartz, I., Conn, A.R., Gould, N., and Toint, P.L.: CUTE: constrained and un-constrained testing environment. ACM Transactions on Mathematical Software21 (1995), 123–160.[2] Fletcher, R.: Practical methods of optimization. John Wiley & Sons, Chichester,1987.[3] Liu, D.C. and Nocedal, J.: On the limited memory BFGS method for large scaleoptimization. Math. Prog. 45 (1989), 503–528.[4] Lukšan, L. and Spedicato, E.: Variable metric methods for unconstrained opti-mization and nonlinear least squares. J. Comput. Appl. Math. 124 (2000), 61–95.[5] Lukšan, L. and Vlček, J.: Sparse and partially separable test problems for un-constrained and equality constrained optimization. Report V-767, ICS AS CR,Prague, 1998.[6] Lukšan, L. and Vlček, J.: A recursive formulation of limited memory variablemetric methods from the Broyden class. Report V-1059, ICS AS CR, Prague,2009.[7] Moughrabi, I.A.: New implicit multistep quasi-Newton methods. NumericalAnalysis and Applications 2 (2009), 154–164.[8] Nocedal, J.: Updating quasi-Newton matrices with limited storage. Math. Comp.35 (1980), 773–782.[9] Vlček, J. and Lukšan, L.: Generalizations of the limited-memory BFGS methodbased on quasi-product form of update. Report V-1060, ICS AS CR, Prague,2009.232

Limited-memory variable metric methods that use quantities from the preceding iteration

http://dml.cz/bitstream/handle/10338.dmlcz/702762/PANM_15-2010-1_32.pdf

Limited-memory variable metric methods that use quantities from the preceding iteration

Abstract

Similar works

Full text

Available Versions

Institute of Mathematics AS CR, v. v. i.