Scalable Algorithms for Parallel Tree-based Adaptive Mesh Refinement with General Element Types

In this thesis, we develop, discuss and implement algorithms for scalable parallel tree-based adaptive mesh refinement (AMR) using space-filling curves (SFCs). We create an AMR software that works independently of the used element type, such as for example lines, triangles, tetrahedra, quadrilaterals, hexahedra, and prisms. For triangular and tetrahedral elements (simplices) with red-refinement (1:4 in 2D, 1:8 in 3D), we develop a new SFC, the tetrahedral Morton space-filling curve (TM-SFC). Its construction is similar to the Morton index for quadrilaterals/hexa- hedra, as it is also based on bitwise interleaving the coordinates of a certain vertex of the simplex, the anchor node. Additionally, we interleave with a new piece of information, the so called type. For these simplices, we develop element local algorithms such as constructing the parent, children, or face-neighbors of a simplex, and show that most of them are constant-time operations independent of the refinement level. With SFC based partitioning it is possible that the mesh elements that are parti- tioned to one process do not form a face-connected domain. We prove the following upper bounds for the number of face-connected components of segments of the TM-SFC: With a maximum refine- ment level of L, the number of face-connected components is bounded by 2(L − 1) in 2D and 2L + 1 in 3D. Additionally, we perform a numerical investigation of the distribution of lengths of SFC segments. Furthermore, we develop a new approach to partition and repartition a coarse (input) mesh among the processes. Compared to previous methods it optimizes for fine mesh load-balance and reduces the parallel communication of coarse mesh data. We discuss the coarse mesh repartitioning algorithm and demonstrate that our method repartitions a coarse mesh of 371e9 trees on 917,504 processes (405,000 trees per process) on the Juqueen supercomputer in 1.2 seconds. We develop an AMR concept that works independently of the element type; achieving this independence by strictly distinguishing between functions that oper- ate on the whole mesh (high-level) and functions that locally operate on a single element or a small set of elements (low-level). We discuss a new approach to generate and manage ghost elements that fits into our element-type independent approach. We define and describe the necessary low-level algorithms. Our main idea is the computation of tree-to-tree face-neighbors of an element via the explicit construction of the element's face as a lower dimensional element. In order to optimize the runtime of this method we enhance the algorithm with a top-down search method from Isaac, Burstedde, Wilcox, and Ghattas, and demonstrate how it speeds up the computation by factors of 10 to 20 achieving runtimes comparable to state-of-the art implementations with fixed element types. With the ghost algorithm we build a straight-forward ripple version of the 2:1 balance algorithm. This is not an optimized version but it serves as a feasibility study for our element-type independent approach. We implement all algorithms that we develop in this thesis in the new AMR library t8code. Our modular approach allows us to reuse existing software, which we demonstrate by using the library p4est for quadrilateral and hexahedral elements. In a concurrent Bachelor's thesis by David Knapp (INS, Bonn) the necessary low-level algorithms for prisms were developed. With t8code we demonstrate that we can create, adapt, (re-)partition, and balance meshes, as well as create and manage a ghost layer. In various tests we show excellent strong and weak scaling behavior of our algorithms on up to 917,504 parallel processes on the Juqueen and Mira supercomputers using up to 858e9 mesh elements. We conclude this thesis by demonstrating how an application can be coupled with the AMR routines. We implement a finite volume based advection solver using t8code and show applications with triangular, quadrilateral, tetrahedral, and hexahedral elements, as well as 2D and 3D hybrid meshes, the latter consisting of tetrahedra, hexahedra, and prisms. Overall, we develop and demonstrate a new simplicial SFC and create a fast and scalable tree-based AMR software that offers a flexibility and generality that was previously not available

Space-Filling Curves Continued: Sierpinski

A new massive-splitting parallelization concept using Sierpinski space-filling curves with dynamic adaptive mesh refinement is presented. We developed a new parallel software concept (independent storage of partitions) for shared-memory systems using a run-length encoded communication and several other beneficial advantages.<br><br>Related work:<br>* Jörn Behrens and Jens Zimmermann, Parallelizing an unstructured grid generator with a space-filling curve approach, in Euro-Par 2000 Parallel Processing<br>* Frank Günther, Miriam Mehl, Markus Pögl, and Christoph Zenger, A cache-aware algorithm for PDEs on hierarchical data structures based on space-filling curves, SIAM Journal on Scientific Computing, 28 (2006)<br>* Michael Bader, Kaveh Rahnema, and Csaba Attila Vigh, Memory-efficient sierpinski-order traversals on dynamically adaptive, recursively structured triangular grids, in Applied Parallel and Scientific Computing<br>* Hans Sagan, Space-filling curves, vol. 18, Springer-Verlag New York, 1994 * David L George, Augmented riemann solvers for the shallow water equations over variable topography with steady states and inundation, Journal of Computational Physics<br

What I’m doing or try to do... Sierpi(nski) & more

Cluster-based parallelization strategy of dynamically adaptive grids using Sierpinski space-filling curve. Content: Cluster-based parallel software concept, refinement and coarsening in each time step, cluster-based optimization strategies, run-length encoded communication methods, possible extensions to SFC cuts, first results on Tsunami simulations on parallel dynamically adaptive grids, etc. <br><br>Related work:<br>* Jörn Behrens and Jens Zimmermann, Parallelizing an unstructured grid generator with a space-filling curve approach, in Euro-Par 2000 Parallel Processing<br>* Frank Günther, Miriam Mehl, Markus Pögl, and Christoph Zenger, A cache-aware algorithm for PDEs on hierarchical data structures based on space-filling curves, SIAM Journal on Scientific Computing, 28 (2006)<br>* Michael Bader, Kaveh Rahnema, and Csaba Attila Vigh, Memory-efficient sierpinski-order traversals on dynamically adaptive, recursively structured triangular grids, in Applied Parallel and Scientific Computing<br>* Hans Sagan, Space-filling curves, vol. 18, Springer-Verlag New York, 1994<br>* David L George, Augmented riemann solvers for the shallow water equations over variable topography with steady states and inundation, Journal of Computational Physics<br

Sierpi Framework

In this talk, we present algorithms for efficient parallel shallow-water simulations on dynamically adaptive grids. We allow multiple partitions per memory context with all data stored independently and use a new run-length encoded concept for efficient data exchange. First of all, this allows OpenMP/TBB parallelization as well as optimizations such as reordering and skipping. We also discuss the MPI data migration strategy with this new parallelization concept. This was presented to Prof. Michael Bader and other members of his group (including Oliver Meister and Kaveh Rahnema) as part of a visit of guest researchers from Hamburg.<br>Related work:<br>* Jörn Behrens and Jens Zimmermann, Parallelizing an unstructured grid generator with a space-filling curve approach, in Euro-Par 2000 Parallel Processing<br>* Frank Günther, Miriam Mehl, Markus Pögl, and Christoph Zenger, A cache-aware algorithm for PDEs on hierarchical data structures based on space-filling curves, SIAM Journal on Scientific Computing, 28 (2006)<br>* Michael Bader, Kaveh Rahnema, and Csaba Attila Vigh, Memory-efficient sierpinski-order traversals on dynamically adaptive, recursively structured triangular grids, in Applied Parallel and Scientific Computing * Hans Sagan, Space-filling curves, vol. 18, Springer-Verlag New York, 1994<br