BFGS-like updates of constraint preconditioners for sequences of KKT linear systems in quadratic programming

We focus on efficient preconditioning techniques for sequences of KKT linear systems arising from the interior point solution of large convex quadratic programming problems. Constraint Preconditioners~(CPs), though very effective in accelerating Krylov methods in the solution of KKT systems, have a very high computational cost in some instances, because their factorization may be the most time-consuming task at each interior point iteration. We overcome this problem by computing the CP from scratch only at selected interior point iterations and by updating the last computed CP at the remaining iterations, via suitable low-rank modifications based on a BFGS-like formula. This work extends the limited-memory preconditioners for symmetric positive definite matrices proposed by Gratton, Sartenaer and Tshimanga in [SIAM J. Optim. 2011; 21(3):912--935, by exploiting specific features of KKT systems and CPs. We prove that the updated preconditioners still belong to the class of exact CPs, thus allowing the use of the conjugate gradient method. Furthermore, they have the property of increasing the number of unit eigenvalues of the preconditioned matrix as compared to generally used CPs. Numerical experiments are reported, which show the effectiveness of our updating technique when the cost for the factorization of the CP is high

Low-rank update of preconditioners for the inexact Newton method with SPD Jacobian

none3In this note preconditioners for the Conjugate Gradient method are studied to solve the Newton system with a symmetric positive definite Jacobian. In particular, we define a sequence of preconditioners built by means of BFGS rank-two updates. Reasonable conditions are derived which guarantee that the preconditioned matrices are not far from the identity in a matrix norm. Some notes on the implementation of the corresponding inexact Newton method are given and some numerical results on a number of model problems illustrate the efficiency of the proposed preconditioners.noneL. BERGAMASCHI; BRU RAFAEL; A. MARTINEZBergamaschi, Luca; Bru, Rafael; MARTINEZ CALOMARDO, Angele

Low-rank updates of balanced incomplete factorization preconditioners

[EN] Let Ax = b be a large and sparse system of linear equations where A is a nonsingular matrix. An approximate solution is frequently obtained by applying preconditioned terations. Consider the matrix B = A + PQT where P,Q ∈ Rn×k are full rank matrices. In this work, we study the problem of updating a previously computed preconditioner for A in order to solve the updated linear system Bx = b by preconditioned iterations. In particular, we propose a method for updating a Balanced Incomplete Factorization preconditioner. The strategy is based on the computation of an approximate Inverse Sherman-Morrison decomposition for an equivalent augmented linear system. Approximation properties of the preconditioned matrix and an analysis of the computational cost of the algorithm are studied. Moreover the results of the numerical experiments with different types of problems show that the proposed method contributes to accelerate the convergence.This work was supported by the Spanish Ministerio de Economia y Competitividad under grant MTM2014-58159-P.Cerdán Soriano, JM.; Marín Mateos-Aparicio, J.; Mas Marí, J. (2017). Low-rank updates of balanced incomplete factorization preconditioners. Numerical Algorithms. 74(2):337-370. https://doi.org/10.1007/s11075-016-0151-6S337370742Bellavia, S., Bertaccini, D., Morini, B.: Nonsymmetric preconditioner updates in Newton-Krylov methods for nonlinear systems. SIAM J. Sci. Comput. 33 (5), 2595–2619 (2011)Benzi, M., Bertaccini, D.: Approximate inverse preconditioning for shifted linear systems. BIT 43(2), 231–244 (2003)Bergamaschi, L., Bru, R., Martínez, A.: Low-rank update of preconditioners for the inexact Newton method with SPD Jacobian. Math. Comput. Model. 54, 1863–1873 (2011)Bergamaschi, L., Bru, R., Martínez, A., Mas, J., Putti, M.: Low-rank update of preconditioners for the nonlinear Richards Equation. Math. Comput. Model. 57, 1933–1941 (2013)Bergamaschi, L., Gondzio, J., Venturin, M., Zilli, G.: Inexact constraint preconditioners for linear systems arising in interior point methods. Comput. Optim. Appl. 36(2-3), 137–147 (2007)Beroiz, M., Hagstrom, T., Lau, S.R., Price, R.H.: Multidomain, sparse, spectral-tau method for helically symmetric flow. Comput. Fluids 102(0), 250–265 (2014)Bertaccini, D.: Efficient preconditioning for sequences of parametric complex symmetric linear systems. Electron. Trans. Numer. Anal. 18, 49–64 (2004)Bollhöfer, M.: A robust and efficient ILU that incorporates the growth of the inverse triangular factors. SIAM J. Sci. Comput. 25(1), 86–103 (2003)Bollhöfer, M., Saad, Y.: On the relations between ILUs and factored approximate inverses. SIAM. J. Matrix Anal. Appl. 24(1), 219–237 (2002)Bru, R., Cerdán, J., Marín, J., Mas, J.: Preconditioning sparse nonsymmetric linear systems with the Sherman-Morrison formula. SIAM J. Sci. Comput. 25(2), 701–715 (2003)Bru, R., Marín, J., Mas, J., Tůma, M.: Balanced incomplete factorization. SIAM J. Sci. Comput. 30(5), 2302–2318 (2008)Bru, R., Marín, J., Mas, J., Tůma, M.: Improved balanced incomplete factorization. SIAM J. Matrix Anal. Appl. 31(5), 2431–2452 (2010)Cerdán, J., Faraj, T., Malla, N., Marín, J., Mas, J.: Block approximate inverse preconditioners for sparse nonsymmetric linear systems. Electron. Trans. Numer. Anal. 37, 23–40 (2010)Cerdán, J., Marín, J., Mas, J., Tůma, M.: Block balanced incomplete factorization. Technical Report No. TR-IMM2015/04, Polytechnic University of Valencia, Spain (2015)Davis, T.A.: University of Florida Sparse Matrix Collection. available online at http://www.cise.ufl.edu/~davis/sparse/ , NA Digest, vol. 94, issue 42, October 1994.Tebbens, J.D., Tůma, M.: Efficient preconditioning of sequences of nonsymmetric linear systems. SIAM J. Sci Comput. 29(5), 1918–1941 (2007)Tebbens, J.D., Tůma, M.: Preconditioner updates for solving sequences of linear systems in matrix-free environment. Numer Linear Algebra Appl. 17, 997–1019 (2010)Embree, M., Sifuentes, J.A., Soodhalter, K.M., Szyld, D.B., Xue, F.: Short-term recurrence Krylov subspace methods for nearly hermitian matrices. SIAM.J. Matrix Anal. Appl. 33-2, 480–500 (2012)Engquist, B., Ying, L.: Sweeping preconditioner for the Helmholtz equation: hierarchical matrix representation. Commun. Pure Appl. Math. 64, 697–735 (2011)Gatto, P., Christiansen, R.E., Hesthaven, J.S.: A preconditioner based on a low-rank approximation with applications to topology optimization. Technical Report EPFL-ARTICLE-207108, École polytechnique fédérale de Lausanne, EPFL, CH-1015 Lausanne, 2015.Grasedyck, L., Hackbusch, W.: Construction and arithmetics of H-matrices. Computing 70(4), 295–334 (2003)Grasedyck, L., Kressner, D., Tobler, C.: A literature survey of low-rank tensor approximation techniques. GAMM-Mitteilungen 36(1), 53–78 (2013)Greengard, L., Rokhlin, V.: A new version of the fast multipole method for the Laplace equation in three dimensions. Acta Numer. 6(1), 229–269 (1997)Hager, W.W.: Updating the inverse of matrix. SIAM Rev. 31(2), 221–239 (1989)Halko, N., Martinsson, P.G., Tropp, J.A.: Finding structure with randomness: probabilistic algorithms for constructing approximate matrix decompositions. SIAM Rev. 53(2), 217–288 (2011)Kelley, C.T.: Solving Nonlinear Equations with Newton’s Method. Fundamentals of algorithms. Society for Industrial and Applied Mathematics (SIAM), Philadelphia (2003)Saad, Y.: ILUT: a dual threshold incomplete L U factorization. Numer. Linear Algebra Appl. 1(4), 387–402 (1994)Saad, Y., Schulz, M.H.: GMRES: a generalized minimal residual algorithm for solving nonsymmetric linear systems. SIAM J. Sci. Stat. Comput. 7, 856–869 (1986)Simoncini, V., Szyld, D.B.: The effect of non-optimal bases on the convergence of Krylov subspace methods. Numer Math. 100(4), 711–733 (2005)van der Vorst, H.A.: Bi-CGSTAB: a fast and smoothly converging variant of bi-CG for the solution of non-symmetric linear systems. SIAM J. Sci. Stat. Comput. 12, 631–644 (1992

Compact quasi-Newton preconditioners for symmetric positive definite linear systems

[EN] In this paper, preconditioners for the conjugate gradient method are studied to solve the Newton system with symmetric positive definite Jacobian. In particular, we define a sequence of preconditioners built by means of Symmetric Rank one (SR1) and Broyden-Fletcher-Goldfarb-Shanno (BFGS) low-rank updates. We develop conditions under which the SR1 update maintains the preconditioner symmetric positive definite. Spectral analysis of the SR1 preconditioned Jacobians shows an improved eigenvalue distribution as the Newton iteration proceeds. A compact matrix formulation of the preconditioner update is developed which reduces the cost of its application and is more suitable to parallel implementation. Some notes on the implementation of the corresponding Inexact Newton method are given and some numerical results on a number of model problems illustrate the efficiency of the proposed preconditioners.This work was supported by the Spanish Ministerio de Economia y Competitividad under grants MTM2014-58159-P, MTM2017-85669-P, and MTM2017-90682-REDT. The first and third authors have been also partially supported by the INdAM Research group GNCS, 2020 Project: Optimization and advanced linear algebra for problems arising from PDEs.Bergamaschi, L.; Marín Mateos-Aparicio, J.; Martinez, A. (2020). Compact quasi-Newton preconditioners for symmetric positive definite linear systems. Numerical Linear Algebra with Applications. 27(6):1-17. https://doi.org/10.1002/nla.2322S117276Bergamaschi, L., & Putti, M. (1999). Mixed finite elements and Newton-type linearizations for the solution of Richards’ equation. International Journal for Numerical Methods in Engineering, 45(8), 1025-1046. doi:10.1002/(sici)1097-0207(19990720)45:83.0.co;2-gNotay, Y. (2001). Combination of Jacobi-Davidson and conjugate gradients for the partial symmetric eigenproblem. Numerical Linear Algebra with Applications, 9(1), 21-44. doi:10.1002/nla.246Bergamaschi, L., & Martínez, A. (2014). Efficiently preconditioned inexact Newton methods for large symmetric eigenvalue problems. Optimization Methods and Software, 30(2), 301-322. doi:10.1080/10556788.2014.908878Bergamaschi, L., & Martínez, A. (2013). Parallel RFSAI-BFGS Preconditioners for Large Symmetric Eigenproblems. Journal of Applied Mathematics, 2013, 1-10. doi:10.1155/2013/767042Martinez, J. M. (1993). A Theory of Secant Preconditioners. Mathematics of Computation, 60(202), 681. doi:10.2307/2153109Morales, J. L., & Nocedal, J. (2000). Automatic Preconditioning by Limited Memory Quasi-Newton Updating. SIAM Journal on Optimization, 10(4), 1079-1096. doi:10.1137/s1052623497327854Nabben, R., & Vuik, C. (2006). A Comparison of Deflation and the Balancing Preconditioner. SIAM Journal on Scientific Computing, 27(5), 1742-1759. doi:10.1137/040608246Freitag, M. A., & Spence, A. (2007). A tuned preconditioner for inexact inverse iteration applied to Hermitian eigenvalue problems. IMA Journal of Numerical Analysis, 28(3), 522-551. doi:10.1093/imanum/drm036Martínez, Á. (2016). Tuned preconditioners for the eigensolution of large SPD matrices arising in engineering problems. Numerical Linear Algebra with Applications, 23(3), 427-443. doi:10.1002/nla.2032Gratton, S., Sartenaer, A., & Tshimanga, J. (2011). On A Class of Limited Memory Preconditioners For Large Scale Linear Systems With Multiple Right-Hand Sides. SIAM Journal on Optimization, 21(3), 912-935. doi:10.1137/08074008Bergamaschi, L., Bru, R., & Martínez, A. (2011). Low-rank update of preconditioners for the inexact Newton method with SPD Jacobian. Mathematical and Computer Modelling, 54(7-8), 1863-1873. doi:10.1016/j.mcm.2010.11.064DeGuchy, O., Erway, J. B., & Marcia, R. F. (2018). Compact representation of the full Broyden class of quasi‐Newton updates. Numerical Linear Algebra with Applications, 25(5). doi:10.1002/nla.2186Nocedal, J., & Wright, S. J. (Eds.). (1999). Numerical Optimization. Springer Series in Operations Research and Financial Engineering. doi:10.1007/b98874Dembo, R. S., Eisenstat, S. C., & Steihaug, T. (1982). Inexact Newton Methods. SIAM Journal on Numerical Analysis, 19(2), 400-408. doi:10.1137/0719025Kelley, C. T. (1999). Iterative Methods for Optimization. doi:10.1137/1.9781611970920Byrd, R. H., Nocedal, J., & Schnabel, R. B. (1994). Representations of quasi-Newton matrices and their use in limited memory methods. Mathematical Programming, 63(1-3), 129-156. doi:10.1007/bf01582063Bergamaschi, L. (2020). A Survey of Low-Rank Updates of Preconditioners for Sequences of Symmetric Linear Systems. Algorithms, 13(4), 100. doi:10.3390/a13040100Powers, R. T., & Størmer, E. (1970). Free states of the canonical anticommutation relations. Communications in Mathematical Physics, 16(1), 1-33. doi:10.1007/bf01645492Ipsen, I. C. F., & Nadler, B. (2009). Refined Perturbation Bounds for Eigenvalues of Hermitian and Non-Hermitian Matrices. SIAM Journal on Matrix Analysis and Applications, 31(1), 40-53. doi:10.1137/070682745Simoncini, V., & Eldén, L. (2002). Bit Numerical Mathematics, 42(1), 159-182. doi:10.1023/a:1021930421106G. Sleijpen, G. L., & Van der Vorst, H. A. (1996). A Jacobi–Davidson Iteration Method for Linear Eigenvalue Problems. SIAM Journal on Matrix Analysis and Applications, 17(2), 401-425. doi:10.1137/s0895479894270427Tapia, R. A., Dennis, J. E., & Schäfermeyer, J. P. (2018). Inverse, Shifted Inverse, and Rayleigh Quotient Iteration as Newton’s Method. SIAM Review, 60(1), 3-55. doi:10.1137/15m104995