SpECTRE: A Task-based Discontinuous Galerkin Code for Relativistic Astrophysics

We introduce a new relativistic astrophysics code, SpECTRE, that combines a discontinuous Galerkin method with a task-based parallelism model. SpECTRE's goal is to achieve more accurate solutions for challenging relativistic astrophysics problems such as core-collapse supernovae and binary neutron star mergers. The robustness of the discontinuous Galerkin method allows for the use of high-resolution shock capturing methods in regions where (relativistic) shocks are found, while exploiting high-order accuracy in smooth regions. A task-based parallelism model allows efficient use of the largest supercomputers for problems with a heterogeneous workload over disparate spatial and temporal scales. We argue that the locality and algorithmic structure of discontinuous Galerkin methods will exhibit good scalability within a task-based parallelism framework. We demonstrate the code on a wide variety of challenging benchmark problems in (non)-relativistic (magneto)-hydrodynamics. We demonstrate the code's scalability including its strong scaling on the NCSA Blue Waters supercomputer up to the machine's full capacity of 22,380 nodes using 671,400 threads.Comment: 41 pages, 13 figures, and 7 tables. Ancillary data contains simulation input file

Task-based adaptive multiresolution for time-space multi-scale reaction-diffusion systems on multi-core architectures

A new solver featuring time-space adaptation and error control has been recently introduced to tackle the numerical solution of stiff reaction-diffusion systems. Based on operator splitting, finite volume adaptive multiresolution and high order time integrators with specific stability properties for each operator, this strategy yields high computational efficiency for large multidimensional computations on standard architectures such as powerful workstations. However, the data structure of the original implementation, based on trees of pointers, provides limited opportunities for efficiency enhancements, while posing serious challenges in terms of parallel programming and load balancing. The present contribution proposes a new implementation of the whole set of numerical methods including Radau5 and ROCK4, relying on a fully different data structure together with the use of a specific library, TBB, for shared-memory, task-based parallelism with work-stealing. The performance of our implementation is assessed in a series of test-cases of increasing difficulty in two and three dimensions on multi-core and many-core architectures, demonstrating high scalability

High Order Cell-Centered Lagrangian-Type Finite Volume Schemes with Time-Accurate Local Time Stepping on Unstructured Triangular Meshes

We present a novel cell-centered direct Arbitrary-Lagrangian-Eulerian (ALE) finite volume scheme on unstructured triangular meshes that is high order accurate in space and time and that also allows for time-accurate local time stepping (LTS). The new scheme uses the following basic ingredients: a high order WENO reconstruction in space on unstructured meshes, an element-local high-order accurate space-time Galerkin predictor that performs the time evolution of the reconstructed polynomials within each element, the computation of numerical ALE fluxes at the moving element interfaces through approximate Riemann solvers, and a one-step finite volume scheme for the time update which is directly based on the integral form of the conservation equations in space-time. The inclusion of the LTS algorithm requires a number of crucial extensions, such as a proper scheduling criterion for the time update of each element and for each node; a virtual projection of the elements contained in the reconstruction stencils of the element that has to perform the WENO reconstruction; and the proper computation of the fluxes through the space-time boundary surfaces that will inevitably contain hanging nodes in time due to the LTS algorithm. We have validated our new unstructured Lagrangian LTS approach over a wide sample of test cases solving the Euler equations of compressible gasdynamics in two space dimensions, including shock tube problems, cylindrical explosion problems, as well as specific tests typically adopted in Lagrangian calculations, such as the Kidder and the Saltzman problem. When compared to the traditional global time stepping (GTS) method, the newly proposed LTS algorithm allows to reduce the number of element updates in a given simulation by a factor that may depend on the complexity of the dynamics, but which can be as large as 4.7.Comment: 31 pages, 13 figure

Lagrangian ADER-WENO Finite Volume Schemes on Unstructured Triangular Meshes Based On Genuinely Multidimensional HLL Riemann Solvers

In this paper we use the genuinely multidimensional HLL Riemann solvers recently developed by Balsara et al. to construct a new class of computationally efficient high order Lagrangian ADER-WENO one-step ALE finite volume schemes on unstructured triangular meshes. A nonlinear WENO reconstruction operator allows the algorithm to achieve high order of accuracy in space, while high order of accuracy in time is obtained by the use of an ADER time-stepping technique based on a local space-time Galerkin predictor. The multidimensional HLL and HLLC Riemann solvers operate at each vertex of the grid, considering the entire Voronoi neighborhood of each node and allows for larger time steps than conventional one-dimensional Riemann solvers. The results produced by the multidimensional Riemann solver are then used twice in our one-step ALE algorithm: first, as a node solver that assigns a unique velocity vector to each vertex, in order to preserve the continuity of the computational mesh; second, as a building block for genuinely multidimensional numerical flux evaluation that allows the scheme to run with larger time steps compared to conventional finite volume schemes that use classical one-dimensional Riemann solvers in normal direction. A rezoning step may be necessary in order to overcome element overlapping or crossing-over. We apply the method presented in this article to two systems of hyperbolic conservation laws, namely the Euler equations of compressible gas dynamics and the equations of ideal classical magneto-hydrodynamics (MHD). Convergence studies up to fourth order of accuracy in space and time have been carried out. Several numerical test problems have been solved to validate the new approach

Hardware-aware block size tailoring on adaptive spacetree grids for shallow water waves.

Spacetrees are a popular formalism to describe dynamically adaptive Cartesian grids. Though they directly yield an adaptive spatial discretisation, i.e. a mesh, it is often more efficient to augment them by regular Cartesian blocks embedded into the spacetree leaves. This facilitates stencil kernels working efficiently on homogeneous data chunks. The choice of a proper block size, however, is delicate. While large block sizes foster simple loop parallelism, vectorisation, and lead to branch-free compute kernels, they bring along disadvantages. Large blocks restrict the granularity of adaptivity and hence increase the memory footprint and lower the numerical-accuracy-per-byte efficiency. Large block sizes also reduce the block-level concurrency that can be used for dynamic load balancing. In the present paper, we therefore propose a spacetree-block coupling that can dynamically tailor the block size to the compute characteristics. For that purpose, we allow different block sizes per spacetree node. Groups of blocks of the same size are identied automatically throughout the simulation iterations, and a predictor function triggers the replacement of these blocks by one huge, regularly rened block. This predictor can pick up hardware characteristics while the dynamic adaptivity of the fine grid mesh is not constrained. We study such characteristics with a state-of-the-art shallow water solver and examine proper block size choices on AMD Bulldozer and Intel Sandy Bridge processors