A Parallel Scalable PETSc-Based Jacobi-Davidson Polynomial Eigensolver with Application in Quantum Dot Simulation

Summary. The Jacobi-Davidson (JD) algorithm recently has gained popularity for finding a few selected interior eigenvalues of large sparse polynomial eigenvalue problems, which commonly appear in many computational science and engineering PDE based applications. As other inner-outer algorithms like Newton type method, the bottleneck of the JD algorithm is to solve approximately the inner correction equation. In the previous work, [Hwang, Wei, Huang, and Wang, A Parallel Additive Schwarz Preconditioned Jacobi-Davidson (ASPJD) Algorithm for Polynomial Eigenvalue Problems in Quantum Dot (QD) Simulation, Journal of Computational Physics (2010)], the authors proposed a parallel restricted additive Schwarz preconditioner in conjunction with a parallel Krylov subspace method to accelerate the convergence of the JD algorithm. Based on the previous computational experiences on the algorithmic parameter tuning for the ASPJD algorithm, we further investigate the parallel performance of a PETSc based ASPJD eigensolver on the Blue Gene/P, and a QD quintic eigenvalue problem is used as an example to demonstrate its scalability by showing the excellent strong scaling up to 2,048 cores

A polynomial Jacobi-Davidson solver with support for non-monomial bases and deflation

[EN] Large-scale polynomial eigenvalue problems can be solved by Krylov methods operating on an equivalent linear eigenproblem (linearization) of size d center dot n where d is the polynomial degree and n is the problem size, or by projection methods that keep the computation in the n-dimensional space. Jacobi-Davidson belongs to the latter class of methods, and, since it is a preconditioned eigensolver, it may be competitive in cases where explicitly computing a matrix factorization is exceedingly expensive. However, a fully fledged implementation of polynomial Jacobi-Davidson has to consider several issues, including deflation to compute more than one eigenpair, use of non-monomial bases for the case of large degree polynomials, and handling of complex eigenvalues when computing in real arithmetic. We discuss these aspects and present computational results of a parallel implementation in the SLEPc library.This work was supported by Agencia Estatal de Investigación (AEI) under Grant TIN2016-75985-P, which includes European Commission ERDF funds.Campos, C.; Jose E. Roman (2020). A polynomial Jacobi-Davidson solver with support for non-monomial bases and deflation. BIT Numerical Mathematics. 60(2):295-318. https://doi.org/10.1007/s10543-019-00778-zS295318602Bai, Z., Su, Y.: SOAR: a second-order Arnoldi method for the solution of the quadratic eigenvalue problem. SIAM J. Matrix Anal. Appl. 26(3), 640–659 (2005)Balay, S., Abhyankar, S., Adams, M., Brown, J., Brune, P., Buschelman, K., Dalcin, L., Eijkhout, V., Gropp, W., Karpeyev, D., Kaushik, D., Knepley, M., May, D., McInnes, L.C., Mills, R., Munson, T., Rupp, K., Sanan, P., Smith, B., Zampini, S., Zhang, H., Zhang, H.: PETSc users manual. Technical report ANL-95/11—revision 3.10, Argonne National Laboratory (2018)Betcke, T., Kressner, D.: Perturbation, extraction and refinement of invariant pairs for matrix polynomials. Linear Algebra Appl. 435(3), 514–536 (2011)Betcke, T., Voss, H.: A Jacobi–Davidson-type projection method for nonlinear eigenvalue problems. Future Gen. Comput. Syst. 20(3), 363–372 (2004)Betcke, T., Higham, N.J., Mehrmann, V., Schröder, C., Tisseur, F.: NLEVP: a collection of nonlinear eigenvalue problems. ACM Trans. Math. Softw. 39(2), 7:1–7:28 (2013)Campos, C., Roman, J.E.: Parallel Krylov solvers for the polynomial eigenvalue problem in SLEPc. SIAM J. Sci. Comput. 38(5), S385–S411 (2016)Effenberger, C.: Robust successive computation of eigenpairs for nonlinear eigenvalue problems. SIAM J. Matrix Anal. Appl. 34(3), 1231–1256 (2013)Effenberger, C., Kressner, D.: Chebyshev interpolation for nonlinear eigenvalue problems. BIT 52(4), 933–951 (2012)Fokkema, D.R., Sleijpen, G.L.G., van der Vorst, H.A.: Jacobi–Davidson style QR and QZ algorithms for the reduction of matrix pencils. SIAM J. Sci. Comput. 20(1), 94–125 (1998)Guo, J.S., Lin, W.W., Wang, C.S.: Numerical solutions for large sparse quadratic eigenvalue problems. Linear Algebra Appl. 225, 57–89 (1995)Hernandez, V., Roman, J.E., Vidal, V.: SLEPc: a scalable and flexible toolkit for the solution of eigenvalue problems. ACM Trans. Math. Softw. 31(3), 351–362 (2005)Higham, N.J., Al-Mohy, A.H.: Computing matrix functions. Acta Numer. 19, 159–208 (2010)Higham, N.J., Mackey, D.S., Tisseur, F.: The conditioning of linearizations of matrix polynomials. SIAM J. Matrix Anal. Appl. 28(4), 1005–1028 (2006)Hochbruck, M., Lochel, D.: A multilevel Jacobi–Davidson method for polynomial PDE eigenvalue problems arising in plasma physics. SIAM J. Sci. Comput. 32(6), 3151–3169 (2010)Hochstenbach, M.E., Sleijpen, G.L.G.: Harmonic and refined Rayleigh–Ritz for the polynomial eigenvalue problem. Numer. Linear Algebra Appl. 15(1), 35–54 (2008)Huang, T.M., Hwang, F.N., Lai, S.H., Wang, W., Wei, Z.H.: A parallel polynomial Jacobi–Davidson approach for dissipative acoustic eigenvalue problems. Comput. Fluids 45(1), 207–214 (2011)Hwang, F.N., Wei, Z.H., Huang, T.M., Wang, W.: A parallel additive Schwarz preconditioned Jacobi–Davidson algorithm for polynomial eigenvalue problems in quantum dot simulation. J.Comput. Phys. 229(8), 2932–2947 (2010)Kressner, D.: A block Newton method for nonlinear eigenvalue problems. Numer. Math. 114, 355–372 (2009)Kressner, D., Roman, J.E.: Memory-efficient Arnoldi algorithms for linearizations of matrix polynomials in Chebyshev basis. Numer. Linear Algebra Appl. 21(4), 569–588 (2014)Lancaster, P.: Linearization of regular matrix polynomials. Electron. J. Linear Algebra 17, 21–27 (2008)Matsuo, Y., Guo, H., Arbenz, P.: Experiments on a parallel nonlinear Jacobi–Davidson algorithm. Procedia Comput. Sci. 29, 565–575 (2014)Meerbergen, K.: Locking and restarting quadratic eigenvalue solvers. SIAM J. Sci. Comput. 22(5), 1814–1839 (2001)Roman, J.E., Campos, C., Romero, E., Tomas, A.: SLEPc users manual. Technical report DSIC-II/24/02—Revision 3.10, D. Sistemes Informàtics i Computació, Universitat Politècnica de València (2018)Romero, E., Roman, J.E.: A parallel implementation of Davidson methods for large-scale eigenvalue problems in SLEPc. ACM Trans. Math. Softw. 40(2), 13:1–13:29 (2014)Rommes, J., Martins, N.: Computing transfer function dominant poles of large-scale second-order dynamical systems. SIAM J. Sci. Comput. 30(4), 2137–2157 (2008)Saad, Y.: Iterative Methods for Sparse Linear Systems, 2nd edn. SIAM Publications, Philadelphia (2003)Sensiau, C., Nicoud, F., van Gijzen, M., van Leeuwen, J.W.: A comparison of solvers for quadratic eigenvalue problems from combustion. Int. J. Numer. Methods Fluids 56(8), 1481–1488 (2008)Sleijpen, G.L.G., van der Vorst, H.A.: A Jacobi–Davidson iteration method for linear eigenvalue problems. SIAM J. Matrix Anal. Appl. 17(2), 401–425 (1996)Sleijpen, G.L.G., Booten, A.G.L., Fokkema, D.R., van der Vorst, H.A.: Jacobi–Davidson type methods for generalized eigenproblems and polynomial eigenproblems. BIT 36(3), 595–633 (1996)Sleijpen, G.L.G., van der Vorst, H.A., Meijerink, E.: Efficient expansion of subspaces in the Jacobi–Davidson method for standard and generalized eigenproblems. Electron. Trans. Numer. Anal. 7, 75–89 (1998)Tisseur, F., Meerbergen, K.: The quadratic eigenvalue problem. SIAM Rev. 43(2), 235–286 (2001)van Gijzen, M.B., Raeven, F.: The parallel computation of the smallest eigenpair of an acoustic problem with damping. Int. J. Numer. Methods Eng. 45(6), 765–777 (1999)van Noorden, T., Rommes, J.: Computing a partial generalized real Schur form using the Jacobi–Davidson method. Numer. Linear Algebra Appl. 14(3), 197–215 (2007)Voss, H.: A Jacobi–Davidson method for nonlinear and nonsymmetric eigenproblems. Comput. Struct. 85(17–18), 1284–1292 (2007

Parallel Krylov Solvers for the Polynomial Eigenvalue Problem in SLEPc

Polynomial eigenvalue problems are often found in scientific computing applications. When the coefficient matrices of the polynomial are large and sparse, usually only a few eigenpairs are required and projection methods are the best choice. We focus on Krylov methods that operate on the companion linearization of the polynomial but exploit the block structure with the aim of being memory-efficient in the representation of the Krylov subspace basis. The problem may appear in the form of a low-degree polynomial (quartic or quintic, say) expressed in the monomial basis, or a high-degree polynomial (coming from interpolation of a nonlinear eigenproblem) expressed in a nonmonomial basis. We have implemented a parallel solver in SLEPc covering both cases that is able to compute exterior as well as interior eigenvalues via spectral transformation. We discuss important issues such as scaling and restart and illustrate the robustness and performance of the solver with some numerical experiments.The first author was supported by the Spanish Ministry of Education, Culture and Sport through an FPU grant with reference AP2012-0608.Campos, C.; Román Moltó, JE. (2016). Parallel Krylov Solvers for the Polynomial Eigenvalue Problem in SLEPc. SIAM Journal on Scientific Computing. 38(5):385-411. https://doi.org/10.1137/15M1022458S38541138

A parallel implementation of Davidson methods for large-scale eigenvalue problems in SLEPc

In the context of large-scale eigenvalue problems, methods of Davidson type such as Jacobi-Davidson can be competitive with respect to other types of algorithms, especially in some particularly difficult situations such as computing interior eigenvalues or when matrix factorization is prohibitive or highly inefficient. However, these types of methods are not generally available in the form of high-quality parallel implementations, especially for the case of non-Hermitian eigenproblems. We present our implementation of various Davidson-type methods in SLEPc, the Scalable Library for Eigenvalue Problem Computations. The solvers incorporate many algorithmic variants for subspace expansion and extraction, and cover a wide range of eigenproblems including standard and generalized, Hermitian and non-Hermitian, with either real or complex arithmetic. We provide performance results on a large battery of test problems.This work was supported by the Spanish Ministerio de Ciencia e Innovacion under project TIN2009-07519. Author's addresses: E. Romero, Institut I3M, Universitat Politecnica de Valencia, Cami de Vera s/n, 46022 Valencia, Spain), and J. E. Roman, Departament de Sistemes Informatics i Computacio, Universitat Politecnica de Valencia, Cami de Vera s/n, 46022 Valencia, Spain; email: [email protected] Alcalde, E.; Román Moltó, JE. (2014). A parallel implementation of Davidson methods for large-scale eigenvalue problems in SLEPc. ACM Transactions on Mathematical Software. 40(2):13:01-13:29. https://doi.org/10.1145/2543696S13:0113:29402P. Arbenz, M. Becka, R. Geus, U. Hetmaniuk, and T. Mengotti. 2006. On a parallel multilevel preconditioned Maxwell eigensolver. Parallel Comput. 32, 2, 157--165.Z. Bai, J. Demmel, J. Dongarra, A. Ruhe, and H. van der Vorst, Eds. 2000. Templates for the Solution of Algebraic Eigenvalue Problems: A Practical Guide. SIAM, Philadelphia, PA.C. G. Baker, U. L. Hetmaniuk, R. B. Lehoucq, and H. K. Thornquist. 2009. Anasazi software for the numerical solution of large-scale eigenvalue problems. ACM Trans. Math. Softw. 36, 3, 13:1--13:23.S. Balay, J. Brown, K. Buschelman, V. Eijkhout, W. Gropp, D. Kaushik, M. Knepley, L. C. McInnes, B. Smith, and H. Zhang. 2011. PETSc users manual. Tech. Rep. ANL-95/11-Revision 3.2, Argonne National Laboratory.S. Balay, W. D. Gropp, L. C. McInnes, and B. F. Smith. 1997. Efficient management of parallelism in object oriented numerical software libraries. In Modern Software Tools in Scientific Computing, E. Arge, A. M. Bruaset, and H. P. Langtangen, Eds., Birkhaüser, 163--202.M. A. Brebner and J. Grad. 1982. Eigenvalues of Ax =λ Bx for real symmetric matrices A and B computed by reduction to a pseudosymmetric form and the HR process. Linear Algebra Appl. 43, 99--118.C. Campos, J. E. Roman, E. Romero, and A. Tomas. 2011. SLEPc users manual. Tech. Rep. DSICII/24/02 - Revision 3.2, D. Sistemes Informàtics i Computació, Universitat Politècnica de València. http://www.grycap.upv.es/slepc.T. Dannert and F. Jenko. 2005. Gyrokinetic simulation of collisionless trapped-electronmode turbulence. Phys. Plasmas 12, 7, 072309.E. R. Davidson. 1975. The iterative calculation of a few of the lowest eigenvalues and corresponding eigenvectors of large real-symmetric matrices. J. Comput. Phys. 17, 1, 87--94.T. A. Davis and Y. Hu. 2011. The University of Florida Sparse Matrix Collection. ACM Trans. Math. Softw. 38, 1, 1:1--1:25.H. C. Elman, A. Ramage, and D. J. Silvester. 2007. Algorithm 866: IFISS, a Matlab toolbox for modelling incompressible flow. ACM Trans. Math. Softw. 33, 2. Article 14.T. Ericsson and A. Ruhe. 1980. The spectral transformation Lanczos method for the numerical solution of large sparse generalized symmetric eigenvalue problems. Math. Comp. 35, 152, 1251--1268.M. Ferronato, C. Janna, and G. Pini. 2012. Efficient parallel solution to large-size sparse eigenproblems with block FSAI preconditioning. Numer. Linear Algebra Appl. 19, 5, 797--815.D. R. Fokkema, G. L. G. Sleijpen, and H. A. van der Vorst. 1998. Jacobi--Davidson style QR and QZ algorithms for the reduction of matrix pencils. SIAM J. Sci. Comput. 20, 1, 94--125.M. A. Freitag and A. Spence. 2007. Convergence theory for inexact inverse iteration applied to the generalised nonsymmetric eigenproblem. Electron. Trans. Numer. Anal. 28, 40--64.M. Genseberger. 2010. Improving the parallel performance of a domain decomposition preconditioning technique in the Jacobi-Davidson method for large scale eigenvalue problems. App. Numer. Math. 60, 11, 1083--1099.V. Hernandez, J. E. Roman, and A. Tomas. 2007. Parallel Arnoldi eigensolvers with enhanced scalability via global communications rearrangement. Parallel Comput. 33, 7--8, 521--540.V. Hernandez, J. E. Roman, and V. Vidal. 2005. SLEPc: A scalable and flexible toolkit for the solution of eigenvalue problems. ACM Trans. Math. Softw. 31, 3, 351--362.V. Heuveline, B. Philippe, and M. Sadkane. 1997. Parallel computation of spectral portrait of large matrices by Davidson type methods. Numer. Algor. 16, 1, 55--75.M. E. Hochstenbach. 2005a. Generalizations of harmonic and refined Rayleigh-Ritz. Electron. Trans. Numer. Anal. 20, 235--252.M. E. Hochstenbach. 2005b. Variations on harmonic Rayleigh--Ritz for standard and generalized eigenproblems. Preprint, Department of Mathematics, Case Western Reserve University.M. E. Hochstenbach and Y. Notay. 2006. The Jacobi--Davidson method. GAMM Mitt. 29, 2, 368--382.F.-N. Hwang, Z.-H. Wei, T.-M. Huang, and W. Wang. 2010. A parallel additive Schwarz preconditioned Jacobi-Davidson algorithm for polynomial eigenvalue problems in quantum dot simulation. J. Comput. Phys. 229, 8, 2932--2947.A. V. Knyazev. 2001. Toward the optimal preconditioned eigensolver: Locally optimal block preconditioned conjugate gradient method. SIAM J. Sci. Comput. 23, 2, 517--541.A. V. Knyazev, M. E. Argentati, I. Lashuk, and E. E. Ovtchinnikov. 2007. Block Locally Optimal Preconditioned Eigenvalue Xolvers (BLOPEX) in HYPRE and PETSc. SIAM J. Sci. Comput. 29, 5, 2224--2239.J. Kopal, M. Rozložník, M. Tuma, and A. Smoktunowicz. 2012. Rounding error analysis of orthogonalization with a non-standard inner product. Numer. Math. 52, 4, 1035--1058.D. Kressner. 2006. Block algorithms for reordering standard and generalized Schur forms. ACM Trans. Math. Softw. 32, 4, 521--532.R. B. Lehoucq, D. C. Sorensen, and C. Yang. 1998. ARPACK Users' Guide, Solution of Large-Scale Eigenvalue Problems by Implicitly Restarted Arnoldi Methods. SIAM, Philadelphia, PA.Z. Li, Y. Saad, and M. Sosonkina. 2003. pARMS: a parallel version of the algebraic recursive multilevel solver. Numer. Linear Algebra Appl. 10, 5--6, 485--509.J. R. McCombs and A. Stathopoulos. 2006. Iterative validation of eigensolvers: a scheme for improving the reliability of Hermitian eigenvalue solvers. SIAM J. Sci. Comput. 28, 6, 2337--2358.F. Merz, C. Kowitz, E. Romero, J. E. Roman, and F. Jenko. 2012. Multi-dimensional gyrokinetic parameter studies based on eigenvalues computations. Comput. Phys. Commun. 183, 4, 922--930.R. B. Morgan. 1990. Davidson's method and preconditioning for generalized eigenvalue problems. J. Comput. Phys. 89, 241--245.R. B. Morgan. 1991. Computing interior eigenvalues of large matrices. Linear Algebra Appl. 154--156, 289--309.R. B. Morgan and D. S. Scott. 1986. Generalizations of Davidson's method for computing eigenvalues of sparse symmetric matrices. SIAM J. Sci. Statist. Comput. 7, 3, 817--825.R. Natarajan and D. Vanderbilt. 1989. A new iterative scheme for obtaining eigenvectors of large, real-symmetric matrices. J. Comput. Phys. 82, 1, 218--228.M. Nool and A. van der Ploeg. 2000. A parallel Jacobi--Davidson-type method for solving large generalized eigenvalue problems in magnetohydrodynamics. SIAM J. Sci. Comput. 22, 1, 95--112.J. Olsen, P. Jørgensen, and J. Simons. 1990. Passing the one-billion limit in full configuration-interaction (FCI) calculations. Chem. Phys. Lett. 169, 6, 463--472.C. C. Paige, B. N. Parlett, and H. A. van der Vorst. 1995. Approximate solutions and eigenvalue bounds from Krylov subspaces. Numer. Linear Algebra Appl. 2, 2, 115--133.E. Romero and J. E. Roman. 2011. Computing subdominant unstable modes of turbulent plasma with a parallel Jacobi--Davidson eigensolver. Concur. Comput.: Pract. Exp. 23, 17, 2179--2191.Y. Saad. 1993. A flexible inner-outer preconditioned GMRES algorithm. SIAM J. Sci. Comput. 14, 2, 461--469.G. L. G. Sleijpen, A. G. L. Booten, D. R. Fokkema, and H. A. van der Vorst. 1996. Jacobi-Davidson type methods for generalized eigenproblems and polynomial eigenproblems. BIT 36, 3, 595--633.G. L. G. Sleijpen and H. A. van der Vorst. 1996. A Jacobi--Davidson iteration method for linear eigenvalue problems. SIAM J. Matrix Anal. Appl. 17, 2, 401--425.G. L. G. Sleijpen and H. A. van der Vorst. 2000. A Jacobi--Davidson iteration method for linear eigenvalue problems. SIAM Rev. 42, 2, 267--293.G. L. G. Sleijpen, H. A. van der Vorst, and E. Meijerink. 1998. Efficient expansion of subspaces in the Jacobi--Davidson method for standard and generalized eigenproblems. Electron. Trans. Numer. Anal. 7, 75--89.A. Stathopoulos. 2007. Nearly optimal preconditioned methods for Hermitian eigenproblems under limited memory. Part I: Seeking one eigenvalue. SIAM J. Sci. Comput. 29, 2, 481--514.A. Stathopoulos and J. R. McCombs. 2007. Nearly optimal preconditioned methods for Hermitian eigenproblems under limited memory. Part II: Seeking many eigenvalues. SIAM J. Sci. Comput. 29, 5, 2162--2188.A. Stathopoulos and J. R. McCombs. 2010. PRIMME: PReconditioned Iterative MultiMethod Eigensolver: Methods and software description. ACM Trans. Math. Softw. 37, 2, 21:1--21:30.A. Stathopoulos and Y. Saad. 1998. Restarting techniques for the (Jacobi-)Davidson symmetric eigenvalue methods. Electron. Trans. Numer. Anal. 7, 163--181.A. Stathopoulos, Y. Saad, and C. F. Fischer. 1995. Robust preconditioning of large, sparse, symmetric eigenvalue problems. J. Comput. Appl. Math. 64, 3, 197--215.A. Stathopoulos, Y. Saad, and K. Wu. 1998. Dynamic thick restarting of the Davidson, and the implicitly restarted Arnoldi methods. SIAM J. Sci. Comput. 19, 1, 227--245.G. W. Stewart. 2001. Matrix Algorithms. Volume II: Eigensystems. SIAM, Philadelphia, PA.H. A. van der Vorst. 2002. Computational methods for large eigenvalue problems. In Handbook of Numerical Analysis, P. G. Ciarlet and J. L. Lions, Eds., Vol. VIII, Elsevier, 3--179.H. A. van der Vorst. 2004. Modern methods for the iterative computation of eigenpairs of matrices of high dimension. Z. Angew. Math. Mech. 84, 7, 444--451.T. van Noorden and J. Rommes 2007. Computing a partial generalized real Schur form using the Jacobi--Davidson method. Numer. Linear Algebra Appl. 14, 3, 197--215.T. D. Young, E. Romero, and J. E. Roman. 2013. Parallel finite element density functional computations exploiting grid refinement and subspace recycling. Comput. Phys. Commun. 184, 1, 66--72

Software for Exascale Computing - SPPEXA 2016-2019

This open access book summarizes the research done and results obtained in the second funding phase of the Priority Program 1648 "Software for Exascale Computing" (SPPEXA) of the German Research Foundation (DFG) presented at the SPPEXA Symposium in Dresden during October 21-23, 2019. In that respect, it both represents a continuation of Vol. 113 in Springer’s series Lecture Notes in Computational Science and Engineering, the corresponding report of SPPEXA’s first funding phase, and provides an overview of SPPEXA’s contributions towards exascale computing in today's sumpercomputer technology. The individual chapters address one or more of the research directions (1) computational algorithms, (2) system software, (3) application software, (4) data management and exploration, (5) programming, and (6) software tools. The book has an interdisciplinary appeal: scholars from computational sub-fields in computer science, mathematics, physics, or engineering will find it of particular interest

A Parallel Scalable PETSc-based Jacobi-Davidson Polynomial Eigensolver with Application in Quantum Dot Simulation

The Jacobi-Davidson (JD) algorithm recently has gained its popularity for finding a few selected interior eigenvalues of large sparse polynomial eigenvalue problems, which commonly appear in many computational science and engineering PDE based applications. As other inner-outer algorithms like Newton type method, the bottleneck of the JD algorithm is to solve approximately the inner correction equation. In the previous work, [Hwang, Wei, Huang, and Wang, A Parallel Additive Schwarz Preconditioned Jacobi-Davidson (ASPJD) Algorithm for Polynomial Eigenvalue Problems in Quantum Dot (QD) Simulation, Journal of Computational Physics, (2010)], the authors proposed a parallel restricted additive Schwarz preconditioner in conjunction with a parallel Krylov subspace method to accelerate the convergence of the JD algorithm. Based on the previous computational experiences on the algorithmic parameter tuning for the ASPJD algorithm, we further investigate the parallel performance of a PETSc based ASPJD eigensolver on the Blue Gene/P, and a QD quintic eigenvalue problem is used as an example to demonstrate its scalability by showing the excellent strong scaling up to 2048 cores

Higher-Order DGFEM Transport Calculations on Polytope Meshes for Massively-Parallel Architectures

In this dissertation, we develop improvements to the discrete ordinates (S_N) neutron transport equation using a Discontinuous Galerkin Finite Element Method (DGFEM) spatial discretization on arbitrary polytope (polygonal and polyhedral) grids compatible for massively-parallel computer architectures. Polytope meshes are attractive for multiple reasons, including their use in other physics communities and their ease in handling local mesh refinement strategies. In this work, we focus on two topical areas of research. First, we discuss higher-order basis functions compatible to solve the DGFEM S_N transport equation on arbitrary polygonal meshes. Second, we assess Diffusion Synthetic Acceleration (DSA) schemes compatible with polytope grids for massively-parallel transport problems. We first utilize basis functions compatible with arbitrary polygonal grids for the DGFEM transport equation. We analyze four different basis functions that have linear completeness on polygons: the Wachspress rational functions, the PWL functions, the mean value coordinates, and the maximum entropy coordinates. We then describe the procedure to extend these polygonal linear basis functions into the quadratic serendipity space of functions. These quadratic basis functions can exactly interpolate monomial functions up to order 2. Both the linear and quadratic sets of basis functions preserve transport solutions in the thick diffusion limit. Maximum convergence rates of 2 and 3 are observed for regular transport solutions for the linear and quadratic basis functions, respectively. For problems that are limited by the regularity of the transport solution, convergence rates of 3/2 (when the solution is continuous) and 1/2 (when the solution is discontinuous) are observed. Spatial Adaptive Mesh Refinement (AMR) achieved superior convergence rates than uniform refinement, even for problems bounded by the solution regularity. We demonstrated accuracy in the AMR solutions by allowing them to reach a level where the ray effects of the angular discretization are realized. Next, we analyzed DSA schemes to accelerate both the within-group iterations as well as the thermal upscattering iterations for multigroup transport problems. Accelerating the thermal upscattering iterations is important for materials (e.g., graphite) with significant thermal energy scattering and minimal absorption. All of the acceleration schemes analyzed use a DGFEM discretization of the diffusion equation that is compatible with arbitrary polytope meshes: the Modified Interior Penalty Method (MIP). MIP uses the same DGFEM discretization as the transport equation. The MIP form is Symmetric Positive De_nite (SPD) and e_ciently solved with Preconditioned Conjugate Gradient (PCG) with Algebraic MultiGrid (AMG) preconditioning. The analysis from previous work was extended to show MIP's stability and robustness for accelerating 3D transport problems. MIP DSA preconditioning was implemented in the Parallel Deterministic Transport (PDT) code at Texas A&M University and linked with the HYPRE suite of linear solvers. Good scalability was numerically verified out to around 131K processors. The fraction of time spent performing DSA operations was small for problems with sufficient work performed in the transport sweep (O(10^3) angular directions). Finally, we have developed a novel methodology to accelerate transport problems dominated by thermal neutron upscattering. Compared to historical upscatter acceleration methods, our method is parallelizable and amenable to massively parallel transport calculations. Speedup factors of about 3-4 were observed with our new method