Closer to the solutions: iterative linear solvers

The solution of dense linear systems received much attention after the second world war, and by the end of the sixties, most of the problems associated with it had been solved. For a long time, Wilkinson's \The Algebraic Eigenvalue Problem" [107], other than the title suggests, became also the standard textbook for the solution of linear systems. When it became clear that partial dierential equations could be solved numerically, to a level of accuracy that was of interest for application areas (such as reservoir engineering, and reactor diusion modeling), there was a strong need for the fast solution of the discretized systems, and iterative methods became popular for these problems

Parallel Iterative Solution Methods for Linear Systems arising from Discretized PDE's

In these notes we will present an overview of a number of related iterative methods for the solution of linear systems of equations. These methods are so-called Krylov projection type methods and the include popular methods as Conjugate Gradients, Bi-Conjugate Gradients, CGST Bi-CGSTAB, QMR, LSQR and GMRES. We will show how these methods can be derived from simple basic iteration formulas. We will not give convergence proofs, but we will refer for these, as far as available, to litterature. Iterative methods are often used in combination with so-called preconditioning operators (approximations for the inverses of the operator of the system to be solved). Since these preconditions are not essential in the derivation of the iterative methods, we will not give much attention to them in these notes. However, in most of the actual iteration schemes, we have included them in order to facilitate the use of these schemes in actual computations. For the application of the iterative schemes one usually thinks of linear sparse systems, e.g., like those arising in the finite element or finite difference approximatious of (systems of) partial differential equations. However, the structure of the operators plays no explicit role in any of these schemes, and these schemes might also successfully be used to solve certain large dense linear systems. Depending on the situation that might be attractive in terms of numbers of floating point operations. It will turn out that all of the iterative are parallelizable in a straight forward manner. However, especially for computers with a memory hierarchy (i.e. like cache or vector registers), and for distributed memory computers, the performance can often be improved significantly through rescheduling of the operations. We will discuss parallel implementations, and occasionally we will report on experimental findings

Augmented Block-Arnoldi Recycling CFD Solvers

One of the limitations of recycled GCRO methods is the large amount of computation required to orthogonalize the basis vectors of the newly generated Krylov subspace for the approximate solution when combined with those of the recycle subspace. Recent advancements in low synchronization Gram-Schmidt and generalized minimal residual algorithms, Swirydowicz et al.~\cite{2020-swirydowicz-nlawa}, Carson et al. \cite{Carson2022}, and Lund \cite{Lund2022}, can be incorporated, thereby mitigating the loss of orthogonality of the basis vectors. An augmented Arnoldi formulation of recycling leads to a matrix decomposition and the associated algorithm can also be viewed as a {\it block} Krylov method. Generalizations of both classical and modified block Gram-Schmidt algorithms have been proposed, Carson et al.~\cite{Carson2022}. Here, an inverse compact $WY$ modified Gram-Schmidt algorithm is applied for the inter-block orthogonalization scheme with a block lower triangular correction matrix $T_k$ at iteration $k$. When combined with a weighted (oblique inner product) projection step, the inverse compact $WY$ scheme leads to significant (over 10$\times$ in certain cases) reductions in the number of solver iterations per linear system. The weight is also interpreted in terms of the angle between restart residuals in LGMRES, as defined by Baker et al.\cite{Baker2005}. In many cases, the recycle subspace eigen-spectrum can substitute for a preconditioner

Domain Decomposition preconditioning for high-frequency Helmholtz problems with absorption

In this paper we give new results on domain decomposition preconditioners for GMRES when computing piecewise-linear finite-element approximations of the Helmholtz equation $-\Delta u - (k^2+ {\rm i} \varepsilon)u = f$, with absorption parameter $\varepsilon \in \mathbb{R}$. Multigrid approximations of this equation with $\varepsilon \not= 0$ are commonly used as preconditioners for the pure Helmholtz case ($\varepsilon = 0$). However a rigorous theory for such (so-called "shifted Laplace") preconditioners, either for the pure Helmholtz equation, or even the absorptive equation ($\varepsilon \not=0$), is still missing. We present a new theory for the absorptive equation that provides rates of convergence for (left- or right-) preconditioned GMRES, via estimates of the norm and field of values of the preconditioned matrix. This theory uses a $k$- and $\varepsilon$-explicit coercivity result for the underlying sesquilinear form and shows, for example, that if $|\varepsilon|\sim k^2$, then classical overlapping additive Schwarz will perform optimally for the absorptive problem, provided the subdomain and coarse mesh diameters are carefully chosen. Extensive numerical experiments are given that support the theoretical results. The theory for the absorptive case gives insight into how its domain decomposition approximations perform as preconditioners for the pure Helmholtz case $\varepsilon = 0$. At the end of the paper we propose a (scalable) multilevel preconditioner for the pure Helmholtz problem that has an empirical computation time complexity of about $\mathcal{O}(n^{4/3})$ for solving finite element systems of size $n=\mathcal{O}(k^3)$, where we have chosen the mesh diameter $h \sim k^{-3/2}$ to avoid the pollution effect. Experiments on problems with $h\sim k^{-1}$, i.e. a fixed number of grid points per wavelength, are also given

Randomized Orthogonal Projection Methods for Krylov Subspace Solvers

Randomized orthogonal projection methods (ROPMs) can be used to speed up the computation of Krylov subspace methods in various contexts. Through a theoretical and numerical investigation, we establish that these methods produce quasi-optimal approximations over the Krylov subspace. Our numerical experiments outline the convergence of ROPMs for all matrices in our test set, with occasional spikes, but overall with a convergence rate similar to that of standard OPMs

The effect of non-optimal bases on the convergence of Krylov subspace methods

There are many examples where non-orthogonality of a basis for Krylov subspace methods arises naturally. These methods usually require less storage or computational effort per iteration than methods using an orthonormal basis (optimal methods), but the convergence may be delayed. Truncated Krylov subspace methods and other examples of non-optimal methods have been shown to converge in many situations, often with small delay, but not in others. We explore the question of what is the effect of having a nonoptimal basis. We prove certain identities for the relative residual gap, i.e., the relative difference between the residuals of the optimal and non-optimal methods. These identities and related bounds provide insight into when the delay is small and convergence is achieved. Further understanding is gained by using a general theory of superlinear convergence recently developed. Our analysis confirms the observed fact that in exact arithmetic the orthogonality of the basis is not important, only the need to maintain linear independence is. Numerical examples illustrate our theoretical results