Abstract: In this paper, we present a class of new generalized quasi-Newton algorithms for unconstrained optimization. The new algorithms are very extensive, including the algorithms in Jiao&apos;s paper and also in Zhangs&apos; even the class of Broyden. The global convergence and the superlinear convergence of the new algorithms are also proved under the weak condition. Numerical experiment indicates that the new algorithms are more feasible and effective

Honglan Zhu

Shu Wen

CiteSeerX

ISSN 1749-3889(print),1749-3897(online)
International Journal of Nonlinear Science
Vol.2(2006) No.3,pp.140-146
A Class of Generalized Quasi-Newton Algorithms with Superlinear
Convergence
Honglan Zhu, Shu Wen 1
Huaiyin Institute of Technology, Huaian, 223003, China
(Received 26 July 2006, accepted 7 September 2006)
Abstract: In this paper, we present a class of new generalized quasi-Newton algorithms for
unconstrained optimization. The new algorithms are very extensive, including the algorithms
in Jiao’s paper and also in Zhangs’ even the class of Broyden. The global convergence and
the superlinear convergence of the new algorithms are also proved under the weak condition.
Numerical experiment indicates that the new algorithms are more feasible and effective.
Keywords: unconstrained optimization; generalized quasi-Newton algorithm; global conver-
gence; superlinear convergence
1 Introduction
Optimization is a very active branch in the computational Mathematics . It is already extensive to apply
in many engineering sections. Readers can refer to[1 ∼ 13] and many other papers.
For unconstrained optimization problem[1]
min f(x), x ∈ Rn, (1.1)
where f(x) is twice continuously differentiable. The standard quasi-Newton equation is
Bk+1δk = yk,, (1.2)
here δk = xk+1 − xk, gk = ∇f(xk), yk = gk+1 − gk and Bk+1 is the approximation of Hessian matrix
G(x) at xk+1 .
Obviously, only two gradients are exploited in (1.2). Techniques using gradients as well as function
values (which are also available) have been studied by several authors. Recently, Zhang[2] made use of the
quadratic function q(τ) = aτ2 + bτ + c, where a, b, c ∈ Rn to approximate the gradient and established a
new quasi-Newton equation
Gk+1δk = yk + γkδk/‖δk‖2 (1.3)
where Gk+1 = ∇2f(xk+1), γk = 3gTk+1δk + 3gTk δk + 6(fk − fk+1). He, furthermore, gave the equivalent
form of (1.3) in another article[3]
Gk+1δk = ŷk (1.4)
where ŷk = yk + γkµk/δTk µk, δ
T
k µk 6= 0, µk is the vector parameter. Zhang proved that ŷk in (1.4) can be
better approximates ∇2f(xk+1) than yk.
Notice that, if we define Rk = fk+1 − fk − gTk δk then we have Rk ≈ 12δTk Gk+1δk for enough small δk.
Thus
1Corresponding author. E-mail address:wenshu119@163.com
Copyright c©World Academic Press, World Academic Union
IJNS.2006.12.15/044
H. Zhu, S. Wen: A Class of Generalized Quasi-Newton Algorithms with Superlinear Convergence 141
δTk Gk+1δk ≈ 2Rk (1.5)
In this paper, we combine (1.4) and (1.5) to the following weighted form
δTk Gk+1δk = θ(δ
T
k yk + γk) + 2(1− θ)Rk, θ ∈ [0, 1].
Then we consider a new extensive equation as follows
δTk Bk+1δk = θ1δ
T
k yk + 2θ2Rk + θ3γk (1.6)
where the scalars θ1, θ2, θ3 ≥ 0, θ1 + θ2 > 0.
Based on (1.6), we deduced a new class of algorithms to solve the optimization problem. Throughout
this paper, we call (1.6) the generalized quasi-Newton equation, and call the algorithms based on (1.6)
generalized quasi-Newton algorithms.
The generalized quasi-Newton equation (1.6) makes use of not only gradient of the objective function,
but also the value of objective function. Thus compared with the standard quasi-Newton equation, the
new algorithms make good use of the information resources. In addition, the generalized quasi-Newton
equation is very extensive. It is the expansion of the standard quasi-Newton equation (1.2). They include
the algorithms in Zhang’s paper[2][3] and also in Jiao’[4] even the class of Broyden. From the above, we can
analyze that the algorithms may be better when θ1 + θ2 = 1 and θ3 = θ1.Actually, numerical experiment at
Section 4 will prove this point.
This paper is organized as follows: we give the generalized quasi-Newton algorithms which are based
on the generalized quasi-Newton equation in Section 2. In Section 3, we prove that the new algorithms have
global convergence and the superlinear convergence. Section 4 presents the numerical experiment.
In the remainder of the paper, we use the following notation: ‖ · ‖denotes the Euclidean norm; x∗ is a
minimizer of f and G∗ which is the Hessian matrix of f atx∗is positive-definite.
2 The Generalized Quasi-Newton Algorithms
In order to obtain the generalized quasi-Newton algorithms, firstly, we consider the rank 1 correction
Bk+1 = Bk + uvT ,
where u, v ∈ Rn.
Let Qk = θ1δTk yk + 2θ2Rk + θ3γk, then from (1.6) we have Qk = δ
T
k Bkδk + δ
T
k uv
T δk. Therefore,
if vT δk 6= 0, then δTk u = 1vT δk (Qk − δ
T
k Bkδk). Let U =
1
vT δk
( Qk
yTk δk
yk − Bkδk), we obtain the rank 1
correction Bk+1 = Bk + 1vT δk (
Qk
yTk δk
yk−Bkδk)vT . Since the Hessian matrix G(x) is symmetric, we require
that its approximate Bk+1 be symmetric also. Therefore, we denote v =
Qk
yTk δk
yk − Bkδk and obtain the
symmetric rank 1 correction formula
Bk+1 = Bk +
1
Qk − δTk Bkδk
(
Qk
yTk δk
yk −Bkδk)( Qk
yTk δk
yk −Bkδk)T .
The weaknesses of rank 1 correction formula are: First, it can’t guarantee that the matrix Bk+1is always
positive-definite; second, it is possible that the numeral is unsteady.
In order to overcome these weaknesses, we directly consider the rank 2 correction which is different
from Jiao[4]. Let
Bk+1 = Bk + auuT + bvvT ,
where the coefficients a, b ∈ R and the vector u, v ∈ Rn.
Exploiting u = 1
vT δk
( Qk
yTk δk
yk −Bkδk), we choose v = yk and obtain
Bk+1 = Bk +
a
(yTk δk)
2
[(
Qk
yTk δk
)2ykyTk −
Qk
yTk δk
(ykδTk Bk + Bkδky
T
k ) + Bkδkδ
T
k Bk] + byky
T
k .
IJNS homepage:http://www.nonlinearscience.org.uk/
142 International Journal of Nonlinear Science,Vol.2(2006),No.3,pp. 140-146
In order to simplify the above formula, we let a = (yTk δk)
2. Because of Bk+1 satisfies (1.6), then we have
b = (Qk−δ
T
k Bkδk)(1−Qk+δTk Bkδk)
(yTk δk)
2 . Therefore, we have the following rank 2 correction formulae
Bk+1 = Bk −
Bkδkδ
T
k Bk
δTk Bkδk
+
Qkyky
T
k
(yTk δk)
2
+ VkV Tk
where Vk = (δTk Bkδk)
1
2 ( yk
yTk δk
− Bkδk
δTk Bkδk
). Since V Tk δk = 0,the formulae still satisfy (1.6) even if the last
term multiplies the constant Φk.
Thus, we obtain a class of rank 2 correction formulae with four parameters
Bk+1 = Bk −
Bkδkδ
T
k Bk
δTk Bkδk
+
Qkyky
T
k
(yTk δk)
2
+ ΦkVkV Tk (2.1)
Obviously, when θ1 = 1, θ2 = θ3 = 0,Φk = 0, (2.1)is the famous class of Broyden. Let Hk = B−1k then
Hk+1 = Hk −
Hkyky
T
k Hk
yTk Hkyk
+
δkδ
T
k
Qk
+ φkzkzTk (2.2)
where zk = (yTk Hkyk)
1
2 ( δk
yTk δk
− Hkyk
yTk Hkyk
) and Hk+1 is the approximation of inverse of Hessian matrix Gk+1.
Here the parameters Φk and φk satisfy[1]
φk =
Φk − 1
Φk(1− µk)− 1 , µk =
δTk Bkδky
T
k Hkyk
(yTk δk)
2
.
Then we present the generalized quasi-Newton algorithms (I)
Step 0: Select an initial point x1 ∈ Rn and the positive-definite matrix B1 ∈ Rn×n or H1 ∈ Rn×n,
choose a sufficiently small constant ε > 0. Let k = 1.
Step 1: If ‖gk‖ = 0, stop; otherwise, go to Step 2.
Step 2: Solve Bkdk = −gkor dk = −Hkgk to get the search direction dk.
Step 3: Starts with λ = 1, find xk+1 = xk +δk, where δk = λkdk, λk > 0 and satisfies the Wolfe search
g(xk + λkdk)T dk ≥ βgTk dk, α < β < 1,
f(xk + λkdk) ≤ f(xk) + αλkgTk dk, 0 ≤ α < 1/2.
Step 4: If ‖xk+1 − xk‖ = 0, stop; otherwise, go to Step 5.
Step 5: Update Bk+1 or Hk+1by formula (2.1) or (2.2), where Qk = θ1δTk yk + 2θ2Rk + θ3γk, and γk
is defined by
γk =
{
3gTk+1δk + 3g
T
k δk + 6(fk − fk+1), if Qk ≥ ε‖δk‖2;
0, otherwise.
Step 6: Let k = k + 1, and go to Step 1.
3 The Convergence Analysis
As the quasi-Newton algorithms are discussed in [1], we can easily prove that the generalized quasi-
Newton algorithms (I) also have the properties of symmetric positive-definite. The algorithms can terminate
in the nth step for quadratic function and not be changed after linear transformation. We state the three
properties as follows.
Theorem 3.1 If Bk is a symmetry positive-definite matrix, and then there is a sufficiently small constant
ε > 0, such that Bk+1 which is updated by (2.1) is symmetric positive-definite.
Theorem 3.2 If f(x) is a quadratic function and G(x) is the Hessian matrix, the following equations
hold with the exact line search
Hi+1yj = δj/(θ1 + θ2), j = 0, 1, · · · , i,
IJNS email for contribution: editor@nonlinearscience.org.uk
H. Zhu, S. Wen: A Class of Generalized Quasi-Newton Algorithms with Superlinear Convergence 143
δTi Gδj = 0, j = 0, 1, · · · , i− 1.
The algorithms (I) must terminate at the m + 1 ≤ niteration; if m + 1 = n, then Hn = G−1.
Theorem 3.3 If (Hk)x = A−1(Hk)yA−T holds for all k, then the algorithms (I) are not changed with
fixed steplength λk after linear transformation y = Ax + a, where A ∈ Rn×n,a ∈ Rn, and (Hk)x,(Hk)y
mean that Hk calculates from the vector x and y, respectively. In order to prove that the algorithms (I) have
the properties of the globe and superlinear convergence which are the main results of this paper, we give the
following assumption which is weaker than [5].
AssumptionH The function f(x)in the problem (1.1) is twice continuously differentiable. It is forcible
convex over the level set D.
Theorem3.4[6] For f(x) is a forcible convex function in the Euclidean space Rn,x0 ∈ Rn. Let
D(x0) = {x ∈ Rn | f(x) ≤ f(x0)},
U∗ = {x ∈ Rn | f(x) ≤ f(z),∀z ∈ Rn},
Then the level set D(x0) is bounded and not empty; both D(x0) and U∗ are closed convex.
Lemma 3.5[6] f(x) is forcible convex over the level set D, if and only if, there is m > 0, such that
m‖z‖2 ≤ zT G(x)z for all z ∈ Rn and all x ∈ D.
From the above Assumption H , Theorem 3.4 and Lemma 3.5, we have G(x) are bounded over the level
set D for all x ∈ D, namely there is m,M > 0, such that m‖z‖2 ≤ zT G(x)z ≤ M‖z‖2 for all z ∈ Rn.
For the arbitrary matrix B ∈ Rn×n, we define
Ψ(B) = tr(B)− ln[det(B)] =
n∑
i=1
(λi − lnλi),
where λ1, λ2, · · · , λn are the eigenvalue of matrix B.
Similar to [5], we can obtain the following Lemma3.6 easily.
Lemma 3.6 If Assumption H holds, then
1) There are the constants C1 > 0, such that
fk+1 − f∗ ≤ (1− αmC1 cos2 ωk)(fk − f∗). (3.1)
where −gTk δk = ‖gk‖ · ‖δk‖ cos ωk, f∗ = f(x∗).
2) There is the constants 0 ≤ C2 < 1, such that
fk+1 − f∗ ≤ Ck2 (f1 − f∗) (3.2)
Lemma 3.7 If Assumption H holds, the scalar Φ ∈ [0, 1). When δk 6= 0, we have
Ψ(Bk+1) ≤ Ψ(Bk) + [M(M
m
)2 − lnm− (1− Φ)] + (1− Φ) ln qk
+(1− Φ)[1− (1− Φ)m
2 − Φ(M2 cos2 ωk + 2Mm cos ωk + m2 cos2 ωk)
(1− Φ)m2 cos2 ωk qk], (3.3)
where M = (θ1 + θ2)M + θ3ε, m = (θ1 + θ2)m− θ3ε and qk = δ
T
k Bkδk
δTk δk
= ‖Bkδk‖‖δk‖ cos ωk.
According to Lemma3.6 and 3.7, we can now prove the globe and superlinear convergence.
Theorem 3.8 If Assumption H holds, x1 and B1 is the initial point and symmetry positive-definite
matrix respectively, Φ ∈ [0, 1), then the sequence xk generated by the algorithms (I) is convergent at x∗.
Proof: When δk 6= 0,Bk can keep positive-definite for k sufficiently large and thus Ψ(Bk) > 0.
Suppose
lim
k→∞
cos2 ωk = 0,
then there is k1 > 0, we have Φcos2 ωk ≤ 12(1− Φ) for all k > k1.
Let Tk =
(1−Φ)−Φcos2 ωk
(1−Φ) cos2 ωk , then Tk >
1
2 cos2 ωk
, hence lnTk > − ln 2−ln cos2 ωk. It is easy to understand
that 1− t ≤ − ln t holds for t > 0. Hence from (3.3) we obtain
IJNS homepage:http://www.nonlinearscience.org.uk/
144 International Journal of Nonlinear Science,Vol.2(2006),No.3,pp. 140-146
Ψ(Bk+1) ≤ Ψ(Bk) + (M − lnm− 1) + (1− Φ) ln qk + (1− Φ)[1− qkTk]
≤ Ψ(Bk) + (M − lnm− 1) + (1− Φ) ln qk + (1− Φ) ln qkTk
≤ Ψ(Bk) + (M − lnm− 1)− (1− Φ) ln Tk
< Ψ(Bk) + (M − lnm− 1) + (1− Φ)(ln 2 + ln cos2 ωk)
Thus, according to Assumption H and the k1, there is k2 > k1 > 0 such that
ln cos2 ωk ≤ − 21− Φ[M − lnm− 1 + (1− Φ) ln 2]
for all k > k2.Thus
Ψ(Bk+1) < Ψ(Bk)− [M − lnm− 1 + (1− Φ) ln 2]
≤ · · · ≤ Ψ(Bk2)− [M − lnM − 1 + (1− Φ) ln 2](k − k2 + 1)
It is easy to find that the term inside the above square brackets is positive. Then the right tends to negative
infinite for the sufficient large k. This is contradict with Ψ(Bk) > 0, so the above supposition can not hold,
namely there is k, such that cos ωk > ε for all k > k, where 0 < ε <
√
1
αmC1
. Substitute it into (3.1), we
have fk+1 − f∗ ≤ (1− αmC1ε2)(fk − f∗). It then follows that the sequence {fk} converges to f∗.
On the other hand, from Assumption H and the Taylor’s theorem we have fk−f∗ ≥ 12m‖xk−x∗‖2 ≥ 0.
Therefore, the sequence {xk} is convergent at x∗.
Theorem 3.9 If Assumption H holds, G∗ = I , then the algorithms(I) satisfies
lim
k→∞
‖(Bk − I)δk‖
‖δk‖ = 0 (3.4)
and λk is taken equal to 1 for sufficient large k, thus the sequence {xk} converges to x∗ superlinearly.
Proof: With Assumption H and the Taylor’s theorem, we can obtain fk+1 − f∗ ≥ 12m‖xk+1 − x∗‖2.
Using this in (3.2), we have ‖xk+1 − x∗‖2 ≤ 2mCk2 (f1 − f∗), then
∞∑
k=0
‖xk+1 − x∗‖ ≤ [ 2
m
(f1 − f∗)]1/2
∞∑
k=0
C
k/2
2 < +∞. (3.5)
From the Proposition 4 in [7], we know (3.4) holds.
Using Theorem 2.2 in [8], we know (3.4) is equivalent to the fact that the sequence xksuperlinearly
converges to x∗.
Next, we will prove λk → 1 (similar to Theorem 6.3 and 6.4 in [9] ). From (3.4), (3.5) and
0 ≤ ‖gk + Gkdk‖‖dk‖ =
‖(Bk −Gk)dk‖
‖dk‖ ≤
‖(Bk −G∗)dk‖
‖dk‖ +
‖(Gk −G∗)dk‖
‖dk‖
≤ ‖(Bk −G∗)δk‖‖δk‖ + M‖xk − x∗‖
we have
lim
k→∞
‖gk + Gkdk‖
‖dk‖ = 0, (3.6)
and hence − gTk dk‖dk‖2 =
dTk Gkdk
‖dk‖2 −
dTk (Gkdk+gk)
‖dk‖2 > 0. Therefore, there is η > 0 such that
−gTk dk ≥ η‖dk‖2 (3.7)
From Theorem 6.3 in [9], namely lim
k→∞
gTk dk
‖dk‖ = 0, we can obtain limk→∞
‖dk‖ = 0. Combining (3.6) and (3.7),
we know that there is uk ∈ (xk, xk + dk) such that the following inequality holds for sufficient large k and
the above η
f(xk + dk)− f(xk)− 12g
T
k dk =
1
2
dTk (G(uk)dk + gk) ≤ (
1
2
− α)η‖dk‖2 ≤ (α− 12)g
T
k dk.
IJNS email for contribution: editor@nonlinearscience.org.uk
H. Zhu, S. Wen: A Class of Generalized Quasi-Newton Algorithms with Superlinear Convergence 145
Hencef(xk + dk)− f(xk) ≤ αgTk dk.
On the other hand, similar to the above proof, we know that there is vk ∈ (xk, xk + dk) such that the
following inequality holds for the sufficient large k and the above η.
dTk g(xk + dk) = d
T
k (g(xk) + G(vk)dk) ≤ ηβ‖dk‖2 ≤ −βdTk g(xk).
Therefore, λk = 1 satisfies the Wolfe search for k sufficiently large.
4 Numerical Experiment
In this section, we will solve the Rosenbrock function which is the typical unconstrained optimization
problem,namely f(x) = 100(x2 − x21)2 + (1− x1)2,where x = (x1, x2)T .
We know x∗ = (1, 1)T and f(x∗) = 0. Let α = 1/4, β = 2/3, ε = 10−15 and Φ = 1/2.
Table 1: The scalars θ1=0.9, θ2=0.5, θ3=0.6
x1 x∗ xJiao∗ xZhang∗(
1.5
0.5
) (
1
1
) (
1.36839237531073
1.87390722131381
) (
0.86160303144629
0.739907870396847
)
Itera. 45 8 6
(
0.5
1.5
)(
1
0.999999999999999
) (
1.01747141620734
1.03532415220092
) (
1.00234389438897
1.00469825826696
)
Itera. 48 8 24
Table 2: The scalars θ1=0.9, θ2=0.5, θ3=0.6
xT1
x∗ xT∗
(0,0) 43 (1,1)
(2,2) 42 (1,0.999999999999999)
(10,10) 54 (1.00000000000001,1.00000000000001)
(100,100) 109 (1,1)
(-10,-1) 65 (0.999999999999999,0.999999999999999)
Table 3: The initial point x1 = (−1.2, 1)T and the scalars θ1 + θ2 = 1, Φ = 0 and θ1 = θ3
θ1 0.1 0.75 0.5 1(θ3 = 0)
x∗
(
1.0000109553952
10000000009543
) (
1.0000000053811
1.0000000008939
) (
1.00000000023
1.00000000446
) (
1
1
)
Time 0.046999999999 0.0469999999997 0.047000000025 0.000109799
Itera. 17 6 9 5
IJNS homepage:http://www.nonlinearscience.org.uk/
146 International Journal of Nonlinear Science,Vol.2(2006),No.3,pp. 140-146
In the tables, x1 is the initial point. x∗, xJiao∗ and x
Zhang
∗ are the results which are solved by the
algorithms in this paper, Jiao’s[4] and Zhang’s[2] paper, respectively. Itera. and Time mean the number and
time of iteration. As reported in Table 1, it is easy to find that our new algorithms behave more efficiently
than Jiao’s and Zhang’s. From Table 2, we can see that the new algorithms in our paper can give us a useful
globe convergence result. In Table 3: for θ1 = 1,θ2 = θ3 = 0 we obtain the BFGS method. Obviously, it
has better performance. Therefore, it is very interesting that we also validate that BFGS method is indeed
one of the best methods up to now.
References
[1] Yaxiang Yuan, Wenyu Sun: Optimization: theory and algorithms. Beijing: Science Press.(2001)
[2] Zhang J Z, Deng NY, Chen L H: New quasi-Newton equation and related methods for unconstrained
optimization. JOTA.102(1),147-167(1999)
[3] Zhang J Z, Xu C X: Properties and numerical performance of quasi-Newton methods with modified
quasi-Newton equation. Journal of Computational Applied Mathematics.137(2), 269-278(2001)
[4] Baocong Jiao: A class of non-quasi-Newton algorithms with superlinear convergence. Numerical
Mathematics a Journal of Chinese University.21(6),178-188(1999)
[5] Broyrd R H, Nocedal J and Yuan Y X: Global convergence of a class of quasi-Newton methods on
convex problems. SIAM Journal on Numerical Analysis.24, 1171-1190(1987)
[6] Shuzhong Shi: Convex analysis. Shanghai: Shanghai scientific & Technical Publishers(1990)
[7] Grienwank, Toint PH L: Local convergence analysis of partitioned quasi-Newton updates. Numerical
Mathematics.39,429-448(1982)
[8] Dennis J E, Moré J J: A characterization of superlinear convergence and its application to quasi-Newton
methods. Mathematical Computing.28,549-560(1971)
[9] Dennis J E, Moré J J: Quasi-Newton methods motivation and theory. SIAM Review. 19,46-89(1977)
[10] Dongdong Wang , Hongbo Hua: On Stability Number of Upper Irredundance Number in Graphs.
International Joural of Nonlinear Science. 2(1),39-42(2006)
[11] Yuchun Wang 1, Lixia Wang, Wenbin Zhang: Application of the Adomian Decomposition Method to
Fully Nonlinear Sine-Gordon Equation. International Joural of Nonlinear Science. 2(1),29-38(2006)
[12] Wenbin Zhang: Attractor of Cauchy Problem in Dissipative KdV Type Equation.International Joural
of Nonlinear Science.1(3),155-163(2006)
[13] Sevdzhan Hakkaev, Kiril Kirchev: On the well-posedness and stability of Peakons for a generalized
Camassa-Holm equation. International Joural of Nonlinear Science. 3(1), 139-148(2006)
IJNS email for contribution: editor@nonlinearscience.org.uk


A Class of Generalized Quasi-Newton Algorithms with Superlinear Convergence

http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=84C04D738F88698CD80F0169753983BF?doi=10.1.1.1036.4005&rep=rep1&type=pdf

A Class of Generalized Quasi-Newton Algorithms with Superlinear Convergence

Abstract

Similar works

Full text

Available Versions

CiteSeerX