Implementation and analysis of a parallel vertex-centered finite element segmental refinement multigrid solver

In a parallel vertex-centered finite element multigrid solver, segmental refinement can be used to avoid all inter-process communication on the fine grids. While domain decomposition methods generally require coupled subdomain processing for the numerical solution to a nonlinear elliptic boundary value problem, segmental refinement exploits that subdomains are almost decoupled with respect to high-frequency error components. This allows to perform multigrid with fully decoupled subdomains on the fine grids, which was proposed as a sequential low-storage algorithm by Brandt in the 1970s, and as a parallel algorithm by Brandt and Diskin in 1994. Adams published the first numerical results from a multilevel segmental refinement solver in 2014, confirming the asymptotic exactness of the scheme for a cell-centered finite volume implementation. We continue Brandt’s and Adams’ research by experimentally investigating the scheme’s accuracy with a vertex-centered finite element segmental refinement solver. We confirm that full multigrid accuracy can be preserved for a few segmental refinement levels, although we observe a different dependency on the segmental refinement parameter space. We show that various strategies for the grid transfers between the finest conventional multigrid level and the segmental refinement subdomains affect the solver accuracy. Scaling results are reported for a Cray XC30 with up to 4096 cores.M.S

An Anisotropic hphp-Adaptation Framework for Ultraweak Discontinuous Petrov-Galerkin Formulations

In this article, we present a three-dimensional anisotropic $hp$-mesh refinement strategy for ultraweak discontinuous Petrov--Galerkin (DPG) formulations with optimal test functions. The refinement strategy utilizes the built-in residual-based error estimator accompanying the DPG discretization. The refinement strategy is a two-step process: (a) use the built-in error estimator to mark and isotropically $hp$-refine elements of the (coarse) mesh to generate a finer mesh; (b) use the reference solution on the finer mesh to compute optimal $h$- and $p$-refinements of the selected elements in the coarse mesh. The process is repeated with coarse and fine mesh being generated in every adaptation cycle, until a prescribed error tolerance is achieved. We demonstrate the performance of the proposed refinement strategy using several numerical examples on hexahedral meshes

Stability Analysis for Electromagnetic Waveguides. Part 1: Acoustic and Homogeneous Electromagnetic Waveguides

In a time-harmonic setting, we show for heterogeneous acoustic and homogeneous electromagnetic wavesguides stability estimates with the stability constant depending linearly on the length $L$ of the waveguide. These stability estimates are used for the analysis of the (ideal) ultraweak (UW) variant of the Discontinuous Petrov Galerkin (DPG) method. For this UW DPG, we show that the stability deterioration with $L$ can be countered by suitably scaling the test norm of the method. We present the ``full envelope approximation'', a UW DPG method based on non-polynomial ansatz functions that allows for treating long waveguides

A scalable hp-adaptive finite element software with applications in fiber optics

In this dissertation, we present a scalable parallel version of hp3D—a finite element (FE) software for analysis and discretization of complex three-dimensional multiphysics applications. The developed software supports hybrid MPI/OpenMP parallelism for large-scale computation on modern manycore architectures. The focus of the effort lies on the development and optimization of the parallel software infrastructure underlying all distributed computation. We discuss the challenges of designing efficient data structures for isotropic and anisotropic hp-adaptive meshes with tetrahedral, hexahedral, prismatic, and pyramidal elements supporting discretization of the exact sequence energy spaces. While the code supports standard Galerkin methods, special emphasis is given to systems arising from discretization with the discontinuous Petrov–Galerkin (DPG) method. The method guarantees discrete stability by employing locally optimal test functions, and it has a built-in error indicator which we exploit to guide mesh adaptivity. In addition to interfacing with third-party packages for various tasks, we have developed our own tools including a parallel nested dissection solver suitable for scalable FE computation of waveguide geometries. We present weak-scaling results with up to 24576 CPU cores and numerical simulations with more than one billion degrees of freedom. The new software capabilities enable solution of challenging wave propagation problems with important applications in acoustics, elastodynamics, and electromagnetics. These applications are difficult to solve in the high-frequency regime because the FE discretization suffers from significant numerical pollution errors that increase with the wavenumber. It is critical to control these errors to obtain a stable and accurate method. We study the pollution effect for waveguide problems with more than 8000 wavelengths in the context of robust DPG FE discretizations for the time-harmonic Maxwell equations. We also discuss adaptive refinement strategies for multi-mode fiber waveguides where the propagating transverse modes must be resolved sufficiently. Our study shows the applicability of the DPG error indicator to this class of problems. Finally, we present both modeling and computational advancements to a unique three-dimensional DPG FE model for the simulation of laser amplification in a fiber amplifier. Fiber laser amplifiers are of interest in communication technology, medical applications, military defense capabilities, and various other fields. Silica fiber amplifiers can achieve high-power operation with great efficiency. At high optical intensities, multi-mode amplifiers suffer from undesired thermal coupling effects which pose a major obstacle in power-scaling of such devices. Our nonlinear 3D vectorial model is based on the time-harmonic Maxwell equations, and it incorporates both amplification via an active dopant and thermal effects via coupling with the heat equation. The model supports co-, counter-, and bi-directional pumping configurations, as well as inhomogeneous and anisotropic material properties. The high-fidelity simulation comes at the cost of a high-order FE discretization with many degrees of freedom per wavelength. To make the computation more feasible, we have developed a novel longitudinal model rescaling, using artificial material parameters with the goal of preserving certain quantities of interest. Numerical tests demonstrate the applicability and utility of this scaled model in the simulation of an ytterbium-doped, step-index fiber amplifier that experiences laser amplification and heating.Computational Science, Engineering, and Mathematic

Scalable DPG Multigrid Solver for Helmholtz Problems: A Study on Convergence

This paper presents a scalable multigrid preconditioner targeting large-scale systems arising from discontinuous Petrov-Galerkin (DPG) discretizations of high-frequency wave operators. This work is built on previously developed multigrid preconditioning techniques of Petrides and Demkowicz (Comput. Math. Appl. 87 (2021) pp. 12-26) and extends the convergence results from $\mathcal{O}(10^7)$ degrees of freedom (DOFs) to $\mathcal{O}(10^9)$ DOFs using a new scalable parallel MPI/OpenMP implementation. Novel contributions of this paper include an alternative definition of coarse-grid systems based on restriction of fine-grid operators, yielding superior convergence results. In the uniform refinement setting, a detailed convergence study is provided, demonstrating h and p robust convergence and linear dependence with respect to the wave frequency. The paper concludes with numerical results on hp-adaptive simulations including a large-scale seismic modeling benchmark problem with high material contrast

Multi-GPU FFT Matvec for Inverse Problems Involving Shift-Invariant Systems

Hessian-based algorithms for the solution to inverse problems typically require many actions of the Hessian matrix on a vector (matvecs). A direct approach is often computationally intractable for problems with high-dimensional parameter fields or expensive-to-evaluate forward models. For systems that exhibit shift-invariance (e.g. autonomous systems) structure in their discretized form, the discretized linear parameter-to-observable (p2o) maps are block Toeplitz matrices. Moreover, considering causality for time-invariant systems the p2o map and its adjoint are lower- and upper-triangular block Toeplitz, respectively. By exploiting this structure, Hessian matrices for these types of systems can be compactly represented and Hessian matvecs can be efficiently computed through scalable multi-GPU FFT matvecs. Compact representation follows directly from the definition of Block Toeplitz matrices. Fast matrix-vector multiplication is achieved by embedding the block Toeplitz matrix within a block circulant matrix which is diagonalized by the Discrete Fourier Transform. The matrix-vector product then becomes an element-wise vector operation in Fourier space. Furthermore, the action of the adjoint p2o map corresponds to simply applying the complex conjugate in Fourier space, eliminating the need to separately store the Fourier-transformed forward and adjoint map. Exploiting the triangular block Toeplitz structure in this way yields memory savings proportional to the number of time steps Nt and a computational speedup of O(Nt/ log(Nt)). In the context of explicit methods that are suitable for GPU-based computation, the number of time steps is typically very large due to the CFL condition, making the savings of the algorithm substantial. We develop a multi-GPU FFT matvec code for Hessians corresponding to block Toeplitz p2o matrices utilizing the cuFFT and NCCL libraries. Our implementation achieves 75-90% of the maximum memory bandwidth on NVIDIA A100 80GB GPUs for all custom GPU kernels — which correspond to memory-bound operations. We also show strong and weak multi-GPU scaling on the Frontera RTX nodes with up to 81 GPUs.Texas Advanced Computing Center (TACC

Elements:Software A Scalable Open-Source hp-Adaptive FE Software for Complex Multiphysics Applications

Computer models can be used to augment, inform, and even replace expensive experimental measurements in science and engineering. However, complex models of engineering applications can quickly exceed computational capability, driving the need for advanced simulation tools. Applications in high-frequency wave simulation--such as submarine sonar (acoustics), fiber optics (electromagnetics), and structural analysis (elastodynamics)--pose a significant challenge for large-scale simulation. This project advances computational modeling capabilities through the development, documentation, and dissemination of a leading-edge simulation software. The effort builds on decades-long research and code development by the investigators and their project collaborators. Distributed as open-source, the software is accessible to the broader scientific community, thereby contributing to fundamental research and education for computer modeling in science and engineering. Furthermore, the project expands the national workforce by training young computational mathematicians at the graduate and postdoctoral levels. The project results are disseminated through conference presentations, workshops and seminars, as well as publications in scientific journals.The hp3D software leverages hybrid MPI/OpenMP parallelism to run efficiently on NSF extreme-scale computing facilities and interfaces with state-of-the-art third-party scientific libraries. In addition to publishing the hp3D code and documentation, this project focuses on the development of a scalable multigrid (MG) solver based on the pre-asymptotically stable discontinuous Petrov-Galerkin (DPG) finite element method. This DPG-MG solver represents a significant advancement in solver technology as 1) the first robust, scalable solver for problems with highly-indefinite operators, such as high-frequency wave propagation; and 2) the first multigrid solver with support for fully anisotropic hp-adaptive hybrid meshes and a reliable built-in error indicator. Serial implementations of the DPG-MG solver have demonstrated near-linear scaling with respect to degrees of freedom in both time and memory; its parallel implementation significantly expands scientific compute capabilities and enables solution of currently intractable problems in 3D wave simulation.</p