When solving large nonsymmetric systems of linear equations with the restarted GMRES algorithm, one is inclined to select a relatively large restart parameter in the hope of mimicking the full GMRES process. Surprisingly, cases exist where small values of the restart parameter yield convergence in fewer iterations than larger values. Here, two simple examples are presented where GMRES(1) converges exactly in three iterations, while GMRES(2) stagnates. One of these examples reveals that GMRES(1) convergence can be extremely sensitive to small changes in the initial residual

Embree, Mark

Czech

Oxford University Research Archive

THE TORTOISE AND THE HARE RESTART GMRESMARK EMBREEAbstrat. When solving large nonsymmetri systems of linear equations with the restartedGMRES algorithm, one is inlined to selet a relatively large restart parameter in the hope ofmimiking the full GMRES proess. Surprisingly, ases exist where small values of the restartparameter yield onvergene in fewer iterations than larger values. Here, two simple examples arepresented where GMRES(1) onverges exatly in three iterations, while GMRES(2) stagnates. Oneof these examples reveals that GMRES(1) onvergene an be extremely sensitive to small hangesin the initial residual.Key words. Restarted GMRES, Krylov subspae methods.AMS subjet lassiations. 65F10, 37N301. Introdution. GMRES is an iterative method for solving large nonsymmetrisystems of linear equations, Ax = b [8℄. Throughout siene and engineering, thisalgorithm and its variants routinely solve problems with millions of degrees of freedom.Its popularity is rooted in an optimality ondition: At the kth iteration, GMRESomputes the solution estimate xk that minimizes the Eulidean norm of the residualrk = Axk   b over a subspae of dimension k,krkk = minp2Pkp(0)=1 kp(A)r0k;(1.1)where Pk denotes those polynomials with degree not exeeding k, and r0 = b Ax0is the initial residual. As eah iteration enlarges the minimizing subspae, the residualnorm dereases monotonially.GMRES optimality omes at a ost, however, sine eah iteration demands bothmore arithmeti and memory than the one before it. A standard work-around isto restart the proess after some xed number of iterations, m. The resulting algo-rithm, GMRES(m), uses the approximate solution xm as the initial guess for a newrun of GMRES, ontinuing this proess until onvergene. The global optimality ofthe original algorithm is lost, so although the residual norms remain monotoni, therestarted proess an stagnate with a non-zero residual, failing to ever onverge [8℄.Sine GMRES(m) enfores loal optimality on m-dimensional spaes, one antiipatesthat inreasing m will yield onvergene in fewer iterations. Many pratial examplesonrm this intuition.We denote the kth residual of GMRES(m) by r(m)k . To be preise, one ylebetween restarts of GMRES(m) is ounted asm individual iterations. Conventionally,then, one expets kr(m)k k  kr(`)k k for ` < m. Indeed, this must be true when k  m.Surprisingly, inreasing the restart parameter sometimes leads to slower onver-gene: kr(m)k k > kr(`)k k for ` < m < k. The author enountered this phenomenonwhile solving a disretized onvetion-diusion equation desribed in [4℄. In unpub-lished experiments, de Sturler [1℄ and Walker and Watson [11℄ observed similar be-havior arising in pratial appliations. One wonders, how muh smaller than kr(m)k kmight kr(`)k k be? The smallest possible ases ompare GMRES(1) to GMRES(2) for3-by-3 matries. Eiermann, Ernst, and Shneider present suh an example for whihOxford University Computing Laboratory, Wolfson Building, Parks Road, Oxford OX1 3QD,United Kingdom (mark.embreeomlab.ox.a.uk). Supported by UK Engineering and Physial Si-enes Researh Counil Grant GR/M12414. 12 MARK EMBREEkr(1)4 k=kr(2)4 k = 0:2154 : : : [2, pp. 284{285℄. Otherwise, the phenomenon we desribehas apparently reeived little attention in the literature.The purpose of this artile is twofold. First, we desribe a pair of extreme ex-amples where GMRES(1) onverges exatly at the third iteration, while GMRES(2)seems to never onverge. The seond example leads to our seond point: Small per-turbations to the initial residual an dramatially alter the onvergene behavior ofGMRES(1).2. First Example. Consider using restarted GMRES to solve Ax = b forA = 0 1 1 10 1 30 0 1 1A; b = 0 2 41 1A:(2.1)Taking x0 = 0 yields the initial residual r0 = b. Using the fat that A and r0 arereal, we an derive expliit formulas for GMRES(1) and GMRES(2) diretly from theGMRES optimality ondition (1.1). The reurrene for GMRES(1),r(1)k+1 = r(1)k   r(1)Tk Ar(1)kr(1)Tk ATAr(1)k Ar(1)k ;(2.2)was studied as early as the 1950s [3, x71℄,[7℄. For the A and r0 = b dened in (2.1),this iteration onverges exatly at the third step:r(1)1 = 0 3 30 1A; r(1)2 = 0 300 1A; r(1)3 = 0 000 1A:Expressions for one GMRES(2) yle an likewise be derived using elementary alu-lus. The updated residual takes the form r(2)k+2 = p(A)r(2)k , where p(z) = 1+z+z2is a quadrati whose oeÆients  = (A; r(2)k ) and  = (A; r(2)k ) are given by = (r(2)Tk AAr(2)k )(r(2)Tk ATAAr(2)k )  (r(2)Tk Ar(2)k )(r(2)Tk ATATAAr(2)k )(r(2)Tk ATAr(2)k )(r(2)Tk ATATAAr(2)k )  (r(2)Tk ATAAr(2)k )(r(2)Tk ATAAr(2)k ) ; = (r(2)Tk Ar(2)k )(r(2)Tk ATAAr(2)k )  (r(2)Tk AAr(2)k )(r(2)Tk ATAr(2)k )(r(2)Tk ATAr(2)k )(r(2)Tk ATATAAr(2)k )  (r(2)Tk ATAAr(2)k )(r(2)Tk ATAAr(2)k ) :Exeuting GMRES(2) on the matrix and right hand side (2.1) revealsr(2)1 = 0 3 30 1A; r(2)2 = 12 0 303 1A; r(2)3 = 128 0 24 2733 1A; r(2)4 = 1122 0 81 108162 1A:The inferiority of GMRES(2) ontinues well beyond the fourth iteration. For example:k kr(2)k k=kr0k5 0.376888290025532.. .10 0.376502488858910.. .15 0.376496927936533.. .20 0.376496055944867.. .25 0.376495995285626.. .30 0.376495984909087.. .RESTARTED GMRES 3kr(m)k kkr0kiteration, kGMRES(1)GMRES(2)0 5 10 15 20 25 3010010 510 1010 15Fig. 1. Convergene urves for GMRES(1) and GMRES(2) applied to (2.1) with x0 = 0.The entire onvergene urve for the rst thirty iterations is shown in Figure 1, basedon performing GMRES(2) in exat arithmeti using Mathematia.The partiular value of b (and thus r0) studied above is exeptional, as it isunusual for GMRES(1) to onverge exatly in three iterations. Remarkably, though,GMRES(1) maintains superiority over GMRES(2) for a wide range of initial residuals.For this matrix A, GMRES(2) onverges exatly in one yle for any initial residualwith zero in the third omponent, so we restrit attention to residuals normalized tothe form r0 = (; ; 1)T. Figure 2 indiates that GMRES(2) makes little progress formost suh residuals, while GMRES(1) onverges to high auray for the vast major-ity of these r0 values. The olor in eah plot reets the magnitude of kr(m)100 k=kr0k:Blue indiates satisfatory onvergene, while red signals little progress in one hun-dred iterations. (To ensure this data's delity, we performed these omputations inboth double and quadruple preision arithmeti; dierenes between the two werenegligible.)To gain an appreiation for the dynamis behind Figure 2, we rst examine theation of a single GMRES(1) step. From (2.2) it is lear that GMRES(1) will om-pletely stagnate only when rT0Ar0 = 0. For the matrix A speied in (2.1) andr0 = (; ; 1)T, this ondition redues to2 +  + 2 +  + 3 + 1 = 0;(2.3)the equation for an oblique ellipse in the (; ) plane.Now writing r(1)k = (; ; 1)T, onsider the map r(1)k 7! s(1)k+1 that projets r(1)k+1into the (; ) plane, s(1)k+1 = (r(1)k+1) 13 0 (r(1)k+1)1(r(1)k+1)21A;4 MARK EMBREE 10  5 0 5 10 10 50510 10  5 0 5 10 10 5051010 1510 1010 5100  Fig. 2. Convergene of GMRES(1) (left) and GMRES(2) (right) for the matrix in (2.1) overa range of initial residuals of the form r0 = (; ; 1)T. The olor indiates kr(m)100 k=kr0k on a loga-rithmi sale: blue regions orrespond to initial residuals that onverge satisfatorily, while the redregions show residuals that stagnate or onverge very slowly.where (r(1)k+1)j denotes the jth entry of r(1)k+1, whih itself is derived from r(1)k via (2.2).For the present example, we haves(1)k+1 = 0BBB  3   42 + 3 + 9   4   12 +  +  + 5 + 103 + 2   32 + 22   2   3 +    32 +  +  + 5 + 10 1CCCA:(2.4)We an lassify the xed points (; ) satisfying (2.3) by investigating the Jaobianof (2.4). One of its eigenvalues is always one, while the other eigenvalue varies aboveand below one in magnitude. In the left plot of Figure 2, we show the stable portionof the ellipse (2.3) in blak and the unstable part in white.We an similarly analyze GMRES(2). This iteration will never progress when, inaddition to the stagnation ondition for GMRES(1), r0 also satises rT0AAr0 = 0.For the present example, this requirement implies2 + 2 + 2 + 5 + 6 + 1 = 0;the equation for an oblique parabola. This urve intersets the ellipse (2.3) at twopoints, drawn as dots in the right plot of Figure 2, the only stagnating residuals(; ; 1)T for GMRES(2). We an analyze their stability as done above for GMRES(1).The projeted map for this iteration, r(2)k 7! s(2)k+2, takes the forms(2)k+2 = 0BB 32   3 + 4 + 9    42   3 + 4 + 9 1CCA:(2.5)Analyzing the Jaobian for this GMRES(2) map at the pair of xed points, we nd oneto be unstable (shown in blak in the right plot of Figure 2) while the other is stable(shown in white). This stable xed point is an attrator for stagnating residuals.RESTARTED GMRES 5kr(m)k kkr0kiteration, kGMRES(1)GMRES(2)0 5 10 15 20 25 3010010 510 1010 15Fig. 3. Convergene urves for GMRES(1) and GMRES(2) applied to (3.1) with x0 = 0.We return briey to the initial residual r0 = (2; 4; 1)T. After the rst few itera-tions, the angle between r(2)k and the xed vetor steadily onverges to zero at therate 0:6452 : : : suggested by the Jaobian's dominant eigenvalue. We onlude withhigh ondene that GMRES(2) never onverges for this initial residual. (If one yleof GMRES(m) produes a residual parallel to r0, then either r(m)m = r0 or r(m)m = 0.Thus a residual an't remain xed in the nite (; ) plane, but still onverge to 0.)3. Seond Example. The matrixA in (2.1) is nondiagonalizable, and one mightbe tempted to blame its surprising onvergene behavior on this fat. To demonstratethat nondiagonalizablity is not an essential requirement, we exhibit a diagonalizablematrix with eigenvalues f1; 2; 3g for whih restarted GMRES also produes extremebehavior. Take A = 0B 1 2  20 2 40 0 3 1CA; b = 0 311 1A;(3.1)with x0 = 0. Again, we onstrut the rst few residuals. For GMRES(1),r(1)1 = 0 2 10 1A; r(1)2 = 0 200 1A; r(1)3 = 0 000 1A;while GMRES(2) yieldsr(2)1 = 0 2 10 1A; r(2)2 = 0 10 1 1A; r(2)3 = 117 0 812 8 1A; r(2)4 = 167 0 1212 28 1A:Figure 3 illustrates the onvergene urve for thirty iterations, again omputed usingexat arithmeti.6 MARK EMBREEAs with the rst example, we investigate the performane of restarted GMRESfor a range of r0 = (; ; 1)T, shown in Figure 4. GMRES(2) performs as before, mak-ing little progress for virtually all the residuals shown; there are two xed points, onestable and the other not. The GMRES(1) phase plane, on the other hand, ontainsfasinating struture. Whether the iteration onverges or stagnates appears sensitivelydependent on the initial residual, highlighted in Figure 5. Red regions|indiatingstagnation|are drawn towards the ar of stable xed points (shown in blak in Fig-ure 4). The boundary between stagnating and onverging residuals exhibits hallmark 10  5 0 5 10 10 50510 10  5 0 5 10 10 5051010 1510 1010 5100  Fig. 4. Comparison of GMRES(1) (left) and GMRES(2) (right) as in Figure 2, but for thematrix in (3.1). (Double and quadruple preision omputations dier notably only at the boundariesbetween onvergene and stagnation.)7:62 7:625 7:63 7:635 7:64 5:29 5:285 5:28 5:275 5:2710 1510 1010 5100Fig. 5. Close-up of the left plot of Figure 4. The white urve denotes unstable xed points ofthe map s(1)k+1.RESTARTED GMRES 7fratal qualities, as one might establish by analyzing the projeted GMRES(1) maps(1)k+1 = 0BBB 23   2   6 + 22   11      32   2 +     + 2   1043   2   42   82   8 + 8 + 7   12 22 + 22   2 + 2   4 + 20 1CCCA:4. Closing Remarks. Examples like (2.1) and (3.1) abound. We disoveredthese by varying the three upper right entries of A and the rst two omponents ofb among the integers from  5 to 5, while xing all other entries. Even among suh arestrited set, numerous other examples exhibit similar behavior.Our ontrived examples are extreme models of a phenomenon experiened inpratial omputations. For the onvetion-diusion disretization desribed in [4℄,GMRES(1) or GMRES(5) an outperform GMRES(20) on moderately rened grids.The optimal hoie of restart parameter depends on the problem. Sine, on average,one GMRES(`) iteration is heaper than one GMRES(m) iteration when ` < m [5, 6℄,the potential advantage of smaller restarts is aute.There is muh more to learn about this unusual restarting behavior. How om-mon is sensitive dependene on r0, espeially for larger restarts? What haraterizessuseptible matries? One hopes an improved GMRES onvergene theory will iden-tify better pratial guidelines for hoosing the restart parameter. One also wondersif related algorithms, inluding GMRES restarted with an augmented subspae [2℄and BiCGSTAB(`) [9℄, exhibit similar unexpeted behavior. Suh eets might alsoarise from automati shift-seletion strategies in the restarted Arnoldi algorithm foralulating eigenvalues [10℄.Aknowledgements. I am indebted to Nik Trefethen and Andy Wathen foradvie that both inuened the ourse of this researh and improved its presentation.I also thank Henk van der Vorst for his helpful omments.REFERENCES[1℄ E. de Sturler, All good things ome to an end, or the onvergene of GMRES(m), July 1997.Talk at SIAM's 45th Anniversary Meeting, Stanford University.[2℄ M. Eiermann, O. G. Ernst, and O. Shneider, Analysis of aeleration strategies forrestarted minimal residual methods, J. Comp. Appl. Math., 123 (2000), pp. 261{292.[3℄ D. K. Faddeev and V. N. Faddeeva, Computational Methods of Linear Algebra, Freeman,San Franiso, 1963. Translated by Robert C. Williams.[4℄ B. Fisher, A. Ramage, D. Silvester, and A. J. Wathen, On parameter hoie and itera-tive onvergene of stabilised advetion-diusion problems, Comput. Methods Appl. Meh.Engrg., 179 (1999), pp. 185{202.[5℄ Y. Huang and H. van der Vorst, Some observations on the onvergene behavior of GMRES,Teh. Rep. 89{09, Faulty of Tehnial Mathematis and Informatis, Delft University ofTehnology, 1989.[6℄ W. A. Joubert, On the onvergene behavior of the restarted GMRES algorithm for solvingnonsymmetri linear systems, Numer. Lin. Alg. Applis., 1 (1994), pp. 427{447.[7℄ M. A. Krasnosel'ski and S. G. Kren, An iteration proess with minimal residuals, Mat.Sbornik N.S., 31(73) (1952), pp. 315{334. (In Russian).[8℄ Y. Saad and M. H. Shultz, GMRES: A generalized minimal residual algorithm for solvingnonsymmetri linear systems, SIAM J. Si. Stat. Comput., 7 (1986), pp. 856{869.[9℄ G. L. G. Sleijpen and D. R. Fokkema, BiCGSTAB(`) for linear equations involving unsym-metri matries with omplex spetrum, Elet. Trans. Numer. Anal., 1 (1993), pp. 11{32.[10℄ D. C. Sorensen, Impliit appliation of polynomial lters in a k-step Arnoldi method, SIAMJ. Matrix Anal. Appl., 13 (1992), pp. 357{385.[11℄ L. T. Watson, personal ommuniation, February 2001.

English

The Tortoise and the Hare restart GMRES

Oxford University Research Archive (ORA)

Mathematical Institute Eprints Archive

THE TORTOISE AND THE HARE RESTART GMRES
MARK EMBREE

Abstrat. When solving large nonsymmetri systems of linear equations with the restarted
GMRES algorithm, one is inlined to selet a relatively large restart parameter in the hope of
mimiking the full GMRES proess. Surprisingly, ases exist where small values of the restart
parameter yield onvergene in fewer iterations than larger values. Here, two simple examples are
presented where GMRES(1) onverges exatly in three iterations, while GMRES(2) stagnates. One
of these examples reveals that GMRES(1) onvergene an be extremely sensitive to small hanges
in the initial residual.
Key words. Restarted GMRES, Krylov subspae methods.
AMS subjet lassiations. 65F10, 37N30
1. Introdution. GMRES is an iterative method for solving large nonsymmetri
systems of linear equations, Ax = b [8℄. Throughout siene and engineering, this
algorithm and its variants routinely solve problems with millions of degrees of freedom.
Its popularity is rooted in an optimality ondition: At the kth iteration, GMRES
omputes the solution estimate x
k
that minimizes the Eulidean norm of the residual
r
k
= Ax
k
  b over a subspae of dimension k,
kr
k
k = min
p2P
k
p(0)=1
kp(A)r
0
k;(1.1)
where P
k
denotes those polynomials with degree not exeeding k, and r
0
= b Ax
0
is the initial residual. As eah iteration enlarges the minimizing subspae, the residual
norm dereases monotonially.
GMRES optimality omes at a ost, however, sine eah iteration demands both
more arithmeti and memory than the one before it. A standard work-around is
to restart the proess after some xed number of iterations, m. The resulting algo-
rithm, GMRES(m), uses the approximate solution x
m
as the initial guess for a new
run of GMRES, ontinuing this proess until onvergene. The global optimality of
the original algorithm is lost, so although the residual norms remain monotoni, the
restarted proess an stagnate with a non-zero residual, failing to ever onverge [8℄.
Sine GMRES(m) enfores loal optimality on m-dimensional spaes, one antiipates
that inreasing m will yield onvergene in fewer iterations. Many pratial examples
onrm this intuition.
We denote the kth residual of GMRES(m) by r
(m)
k
. To be preise, one yle
between restarts of GMRES(m) is ounted asm individual iterations. Conventionally,
then, one expets kr
(m)
k
k  kr
(`)
k
k for ` < m. Indeed, this must be true when k  m.
Surprisingly, inreasing the restart parameter sometimes leads to slower onver-
gene: kr
(m)
k
k > kr
(`)
k
k for ` < m < k. The author enountered this phenomenon
while solving a disretized onvetion-diusion equation desribed in [4℄. In unpub-
lished experiments, de Sturler [1℄ and Walker and Watson [11℄ observed similar be-
havior arising in pratial appliations. One wonders, how muh smaller than kr
(m)
k
k
might kr
(`)
k
k be? The smallest possible ases ompare GMRES(1) to GMRES(2) for
3-by-3 matries. Eiermann, Ernst, and Shneider present suh an example for whih

Oxford University Computing Laboratory, Wolfson Building, Parks Road, Oxford OX1 3QD,
United Kingdom (mark.embreeomlab.ox.a.uk). Supported by UK Engineering and Physial Si-
enes Researh Counil Grant GR/M12414.
1
2 MARK EMBREE
kr
(1)
4
k=kr
(2)
4
k = 0:2154 : : : [2, pp. 284{285℄. Otherwise, the phenomenon we desribe
has apparently reeived little attention in the literature.
The purpose of this artile is twofold. First, we desribe a pair of extreme ex-
amples where GMRES(1) onverges exatly at the third iteration, while GMRES(2)
seems to never onverge. The seond example leads to our seond point: Small per-
turbations to the initial residual an dramatially alter the onvergene behavior of
GMRES(1).
2. First Example. Consider using restarted GMRES to solve Ax = b for
A =
0

1 1 1
0 1 3
0 0 1
1
A
; b =
0

2
 4
1
1
A
:(2.1)
Taking x
0
= 0 yields the initial residual r
0
= b. Using the fat that A and r
0
are
real, we an derive expliit formulas for GMRES(1) and GMRES(2) diretly from the
GMRES optimality ondition (1.1). The reurrene for GMRES(1),
r
(1)
k+1
= r
(1)
k
 
r
(1)T
k
Ar
(1)
k
r
(1)T
k
A
T
Ar
(1)
k
Ar
(1)
k
;(2.2)
was studied as early as the 1950s [3, x71℄,[7℄. For the A and r
0
= b dened in (2.1),
this iteration onverges exatly at the third step:
r
(1)
1
=
0

3
 3
0
1
A
; r
(1)
2
=
0

3
0
0
1
A
; r
(1)
3
=
0

0
0
0
1
A
:
Expressions for one GMRES(2) yle an likewise be derived using elementary alu-
lus. The updated residual takes the form r
(2)
k+2
= p(A)r
(2)
k
, where p(z) = 1+z+z
2
is a quadrati whose oeÆients  = (A; r
(2)
k
) and  = (A; r
(2)
k
) are given by
 =
(r
(2)T
k
AAr
(2)
k
)(r
(2)T
k
A
T
AAr
(2)
k
)  (r
(2)T
k
Ar
(2)
k
)(r
(2)T
k
A
T
A
T
AAr
(2)
k
)
(r
(2)T
k
A
T
Ar
(2)
k
)(r
(2)T
k
A
T
A
T
AAr
(2)
k
)  (r
(2)T
k
A
T
AAr
(2)
k
)(r
(2)T
k
A
T
AAr
(2)
k
)
;
 =
(r
(2)T
k
Ar
(2)
k
)(r
(2)T
k
A
T
AAr
(2)
k
)  (r
(2)T
k
AAr
(2)
k
)(r
(2)T
k
A
T
Ar
(2)
k
)
(r
(2)T
k
A
T
Ar
(2)
k
)(r
(2)T
k
A
T
A
T
AAr
(2)
k
)  (r
(2)T
k
A
T
AAr
(2)
k
)(r
(2)T
k
A
T
AAr
(2)
k
)
:
Exeuting GMRES(2) on the matrix and right hand side (2.1) reveals
r
(2)
1
=
0

3
 3
0
1
A
; r
(2)
2
=
1
2
0

3
0
3
1
A
; r
(2)
3
=
1
28
0

24
 27
33
1
A
; r
(2)
4
=
1
122
0

81
 108
162
1
A
:
The inferiority of GMRES(2) ontinues well beyond the fourth iteration. For example:
k kr
(2)
k
k=kr
0
k
5 0.376888290025532.. .
10 0.376502488858910.. .
15 0.376496927936533.. .
20 0.376496055944867.. .
25 0.376495995285626.. .
30 0.376495984909087.. .
RESTARTED GMRES 3
kr
(m)
k
k
kr
0
k
iteration, k
GMRES(1)
GMRES(2)
0 5 10 15 20 25 30
10
0
10
 5
10
 10
10
 15
Fig. 1. Convergene urves for GMRES(1) and GMRES(2) applied to (2.1) with x
0
= 0.
The entire onvergene urve for the rst thirty iterations is shown in Figure 1, based
on performing GMRES(2) in exat arithmeti using Mathematia.
The partiular value of b (and thus r
0
) studied above is exeptional, as it is
unusual for GMRES(1) to onverge exatly in three iterations. Remarkably, though,
GMRES(1) maintains superiority over GMRES(2) for a wide range of initial residuals.
For this matrix A, GMRES(2) onverges exatly in one yle for any initial residual
with zero in the third omponent, so we restrit attention to residuals normalized to
the form r
0
= (; ; 1)
T
. Figure 2 indiates that GMRES(2) makes little progress for
most suh residuals, while GMRES(1) onverges to high auray for the vast major-
ity of these r
0
values. The olor in eah plot reets the magnitude of kr
(m)
100
k=kr
0
k:
Blue indiates satisfatory onvergene, while red signals little progress in one hun-
dred iterations. (To ensure this data's delity, we performed these omputations in
both double and quadruple preision arithmeti; dierenes between the two were
negligible.)
To gain an appreiation for the dynamis behind Figure 2, we rst examine the
ation of a single GMRES(1) step. From (2.2) it is lear that GMRES(1) will om-
pletely stagnate only when r
T
0
Ar
0
= 0. For the matrix A speied in (2.1) and
r
0
= (; ; 1)
T
, this ondition redues to

2
+  + 
2
+  + 3 + 1 = 0;(2.3)
the equation for an oblique ellipse in the (; ) plane.
Now writing r
(1)
k
= (; ; 1)
T
, onsider the map r
(1)
k
7! s
(1)
k+1
that projets r
(1)
k+1
into the (; ) plane,
s
(1)
k+1
= (r
(1)
k+1
)
 1
3
0

(r
(1)
k+1
)
1
(r
(1)
k+1
)
2
1
A
;
4 MARK EMBREE
 10  5 0 5 10
 10
 5
0
5
10
 10  5 0 5 10
 10
 5
0
5
10
10
 15
10
 10
10
 5
10
0
 
 
Fig. 2. Convergene of GMRES(1) (left) and GMRES(2) (right) for the matrix in (2.1) over
a range of initial residuals of the form r
0
= (; ; 1)
T
. The olor indiates kr
(m)
100
k=kr
0
k on a loga-
rithmi sale: blue regions orrespond to initial residuals that onverge satisfatorily, while the red
regions show residuals that stagnate or onverge very slowly.
where (r
(1)
k+1
)
j
denotes the jth entry of r
(1)
k+1
, whih itself is derived from r
(1)
k
via (2.2).
For the present example, we have
s
(1)
k+1
=
0
B
B
B

 
3
  4
2
+ 3 + 9   4   1

2
+  +  + 5 + 10

3
+ 
2
  3
2
+ 2
2
  2   3 +    3

2
+  +  + 5 + 10
1
C
C
C
A
:(2.4)
We an lassify the xed points (; ) satisfying (2.3) by investigating the Jaobian
of (2.4). One of its eigenvalues is always one, while the other eigenvalue varies above
and below one in magnitude. In the left plot of Figure 2, we show the stable portion
of the ellipse (2.3) in blak and the unstable part in white.
We an similarly analyze GMRES(2). This iteration will never progress when, in
addition to the stagnation ondition for GMRES(1), r
0
also satises r
T
0
AAr
0
= 0.
For the present example, this requirement implies

2
+ 2 + 
2
+ 5 + 6 + 1 = 0;
the equation for an oblique parabola. This urve intersets the ellipse (2.3) at two
points, drawn as dots in the right plot of Figure 2, the only stagnating residuals
(; ; 1)
T
for GMRES(2). We an analyze their stability as done above for GMRES(1).
The projeted map for this iteration, r
(2)
k
7! s
(2)
k+2
, takes the form
s
(2)
k+2
=
0
B
B

3

2
  3 + 4 + 9
    4

2
  3 + 4 + 9
1
C
C
A
:(2.5)
Analyzing the Jaobian for this GMRES(2) map at the pair of xed points, we nd one
to be unstable (shown in blak in the right plot of Figure 2) while the other is stable
(shown in white). This stable xed point is an attrator for stagnating residuals.
RESTARTED GMRES 5
kr
(m)
k
k
kr
0
k
iteration, k
GMRES(1)
GMRES(2)
0 5 10 15 20 25 30
10
0
10
 5
10
 10
10
 15
Fig. 3. Convergene urves for GMRES(1) and GMRES(2) applied to (3.1) with x
0
= 0.
We return briey to the initial residual r
0
= (2; 4; 1)
T
. After the rst few itera-
tions, the angle between r
(2)
k
and the xed vetor steadily onverges to zero at the
rate 0:6452 : : : suggested by the Jaobian's dominant eigenvalue. We onlude with
high ondene that GMRES(2) never onverges for this initial residual. (If one yle
of GMRES(m) produes a residual parallel to r
0
, then either r
(m)
m
= r
0
or r
(m)
m
= 0.
Thus a residual an't remain xed in the nite (; ) plane, but still onverge to 0.)
3. Seond Example. The matrixA in (2.1) is nondiagonalizable, and one might
be tempted to blame its surprising onvergene behavior on this fat. To demonstrate
that nondiagonalizablity is not an essential requirement, we exhibit a diagonalizable
matrix with eigenvalues f1; 2; 3g for whih restarted GMRES also produes extreme
behavior. Take
A =
0
B

1 2  2
0 2 4
0 0 3
1
C
A
; b =
0

3
1
1
1
A
;(3.1)
with x
0
= 0. Again, we onstrut the rst few residuals. For GMRES(1),
r
(1)
1
=
0

2
 1
0
1
A
; r
(1)
2
=
0

2
0
0
1
A
; r
(1)
3
=
0

0
0
0
1
A
;
while GMRES(2) yields
r
(2)
1
=
0

2
 1
0
1
A
; r
(2)
2
=
0

1
0
 1
1
A
; r
(2)
3
=
1
17
0

8
12
 8
1
A
; r
(2)
4
=
1
67
0

 12
12
 28
1
A
:
Figure 3 illustrates the onvergene urve for thirty iterations, again omputed using
exat arithmeti.
6 MARK EMBREE
As with the rst example, we investigate the performane of restarted GMRES
for a range of r
0
= (; ; 1)
T
, shown in Figure 4. GMRES(2) performs as before, mak-
ing little progress for virtually all the residuals shown; there are two xed points, one
stable and the other not. The GMRES(1) phase plane, on the other hand, ontains
fasinating struture. Whether the iteration onverges or stagnates appears sensitively
dependent on the initial residual, highlighted in Figure 5. Red regions|indiating
stagnation|are drawn towards the ar of stable xed points (shown in blak in Fig-
ure 4). The boundary between stagnating and onverging residuals exhibits hallmark
 10  5 0 5 10
 10
 5
0
5
10
 10  5 0 5 10
 10
 5
0
5
10
10
 15
10
 10
10
 5
10
0
 
 
Fig. 4. Comparison of GMRES(1) (left) and GMRES(2) (right) as in Figure 2, but for the
matrix in (3.1). (Double and quadruple preision omputations dier notably only at the boundaries
between onvergene and stagnation.)
7:62 7:625 7:63 7:635 7:64
 5:29
 5:285
 5:28
 5:275
 5:27
10
 15
10
 10
10
 5
10
0


Fig. 5. Close-up of the left plot of Figure 4. The white urve denotes unstable xed points of
the map s
(1)
k+1
.
RESTARTED GMRES 7
fratal qualities, as one might establish by analyzing the projeted GMRES(1) map
s
(1)
k+1
=
0
B
B
B

2
3
  
2
  6 + 2
2
  11      3

2
  
2
+     + 2   10
4
3
  
2
   4
2
  8
2
  8 + 8 + 7   12
 2
2
+ 2
2
  2 + 2   4 + 20
1
C
C
C
A
:
4. Closing Remarks. Examples like (2.1) and (3.1) abound. We disovered
these by varying the three upper right entries of A and the rst two omponents of
b among the integers from  5 to 5, while xing all other entries. Even among suh a
restrited set, numerous other examples exhibit similar behavior.
Our ontrived examples are extreme models of a phenomenon experiened in
pratial omputations. For the onvetion-diusion disretization desribed in [4℄,
GMRES(1) or GMRES(5) an outperform GMRES(20) on moderately rened grids.
The optimal hoie of restart parameter depends on the problem. Sine, on average,
one GMRES(`) iteration is heaper than one GMRES(m) iteration when ` < m [5, 6℄,
the potential advantage of smaller restarts is aute.
There is muh more to learn about this unusual restarting behavior. How om-
mon is sensitive dependene on r
0
, espeially for larger restarts? What haraterizes
suseptible matries? One hopes an improved GMRES onvergene theory will iden-
tify better pratial guidelines for hoosing the restart parameter. One also wonders
if related algorithms, inluding GMRES restarted with an augmented subspae [2℄
and BiCGSTAB(`) [9℄, exhibit similar unexpeted behavior. Suh eets might also
arise from automati shift-seletion strategies in the restarted Arnoldi algorithm for
alulating eigenvalues [10℄.
Aknowledgements. I am indebted to Nik Trefethen and Andy Wathen for
advie that both inuened the ourse of this researh and improved its presentation.
I also thank Henk van der Vorst for his helpful omments.
REFERENCES
[1℄ E. de Sturler, All good things ome to an end, or the onvergene of GMRES(m), July 1997.
Talk at SIAM's 45th Anniversary Meeting, Stanford University.
[2℄ M. Eiermann, O. G. Ernst, and O. Shneider, Analysis of aeleration strategies for
restarted minimal residual methods, J. Comp. Appl. Math., 123 (2000), pp. 261{292.
[3℄ D. K. Faddeev and V. N. Faddeeva, Computational Methods of Linear Algebra, Freeman,
San Franiso, 1963. Translated by Robert C. Williams.
[4℄ B. Fisher, A. Ramage, D. Silvester, and A. J. Wathen, On parameter hoie and itera-
tive onvergene of stabilised advetion-diusion problems, Comput. Methods Appl. Meh.
Engrg., 179 (1999), pp. 185{202.
[5℄ Y. Huang and H. van der Vorst, Some observations on the onvergene behavior of GMRES,
Teh. Rep. 89{09, Faulty of Tehnial Mathematis and Informatis, Delft University of
Tehnology, 1989.
[6℄ W. A. Joubert, On the onvergene behavior of the restarted GMRES algorithm for solving
nonsymmetri linear systems, Numer. Lin. Alg. Applis., 1 (1994), pp. 427{447.
[7℄ M. A. Krasnosel'ski

 and S. G. Kre

n, An iteration proess with minimal residuals, Mat.
Sbornik N.S., 31(73) (1952), pp. 315{334. (In Russian).
[8℄ Y. Saad and M. H. Shultz, GMRES: A generalized minimal residual algorithm for solving
nonsymmetri linear systems, SIAM J. Si. Stat. Comput., 7 (1986), pp. 856{869.
[9℄ G. L. G. Sleijpen and D. R. Fokkema, BiCGSTAB(`) for linear equations involving unsym-
metri matries with omplex spetrum, Elet. Trans. Numer. Anal., 1 (1993), pp. 11{32.
[10℄ D. C. Sorensen, Impliit appliation of polynomial lters in a k-step Arnoldi method, SIAM
J. Matrix Anal. Appl., 13 (1992), pp. 357{385.
[11℄ L. T. Watson, personal ommuniation, February 2001.


The Tortoise and the Hare restart GMRES

Abstract

Similar works

Full text

Available Versions

Oxford University Research Archive

Oxford University Research Archive (ORA)

Mathematical Institute Eprints Archive