Asynchronous communication in spectral-element and discontinuous Galerkin methods for atmospheric dynamics – a case study using the High-Order Methods Modeling Environment (HOMME-homme_dg_branch)

The scalability of computational applications on current and next-generation supercomputers is increasingly limited by the cost of inter-process communication. We implement non-blocking asynchronous communication in the High-Order Methods Modeling Environment for the time integration of the hydrostatic fluid equations using both the spectral-element and discontinuous Galerkin methods. This allows the overlap of computation with communication, effectively hiding some of the costs of communication. A novel detail about our approach is that it provides some data movement to be performed during the asynchronous communication even in the absence of other computations. This method produces significant performance and scalability gains in large-scale simulations.publishedVersio

The ESCAPE project : Energy-efficient Scalable Algorithms for Weather Prediction at Exascale

In the simulation of complex multi-scale flows arising in weather and climate modelling, one of the biggest challenges is to satisfy strict service requirements in terms of time to solution and to satisfy budgetary constraints in terms of energy to solution, without compromising the accuracy and stability of the application. These simulations require algorithms that minimise the energy footprint along with the time required to produce a solution, maintain the physically required level of accuracy, are numerically stable, and are resilient in case of hardware failure. The European Centre for Medium-Range Weather Forecasts (ECMWF) led the ESCAPE (Energy-efficient Scalable Algorithms for Weather Prediction at Exascale) project, funded by Horizon 2020 (H2020) under the FET-HPC (Future and Emerging Technologies in High Performance Computing) initiative. The goal of ESCAPE was to develop a sustainable strategy to evolve weather and climate prediction models to next-generation computing technologies. The project partners incorporate the expertise of leading European regional forecasting consortia, university research, experienced high-performance computing centres, and hardware vendors. This paper presents an overview of the ESCAPE strategy: (i) identify domain-specific key algorithmic motifs in weather prediction and climate models (which we term Weather & Climate Dwarfs), (ii) categorise them in terms of computational and communication patterns while (iii) adapting them to different hardware architectures with alternative programming models, (iv) analyse the challenges in optimising, and (v) find alternative algorithms for the same scheme. The participating weather prediction models are the following: IFS (Integrated Forecasting System); ALARO, a combination of AROME (Application de la Recherche a l'Operationnel a Meso-Echelle) and ALADIN (Aire Limitee Adaptation Dynamique Developpement International); and COSMO-EULAG, a combination of COSMO (Consortium for Small-scale Modeling) and EULAG (Eulerian and semi-Lagrangian fluid solver). For many of the weather and climate dwarfs ESCAPE provides prototype implementations on different hardware architectures (mainly Intel Skylake CPUs, NVIDIA GPUs, Intel Xeon Phi, Optalysys optical processor) with different programming models. The spectral transform dwarf represents a detailed example of the co-design cycle of an ESCAPE dwarf. The dwarf concept has proven to be extremely useful for the rapid prototyping of alternative algorithms and their interaction with hardware; e.g. the use of a domain-specific language (DSL). Manual adaptations have led to substantial accelerations of key algorithms in numerical weather prediction (NWP) but are not a general recipe for the performance portability of complex NWP models. Existing DSLs are found to require further evolution but are promising tools for achieving the latter. Measurements of energy and time to solution suggest that a future focus needs to be on exploiting the simultaneous use of all available resources in hybrid CPU-GPU arrangements

Enhancing speed and scalability of the ParFlow simulation code

Regional hydrology studies are often supported by high resolution simulations of subsurface flow that require expensive and extensive computations. Efficient usage of the latest high performance parallel computing systems becomes a necessity. The simulation software ParFlow has been demonstrated to meet this requirement and shown to have excellent solver scalability for up to 16,384 processes. In the present work we show that the code requires further enhancements in order to fully take advantage of current petascale machines. We identify ParFlow's way of parallelization of the computational mesh as a central bottleneck. We propose to reorganize this subsystem using fast mesh partition algorithms provided by the parallel adaptive mesh refinement library p4est. We realize this in a minimally invasive manner by modifying selected parts of the code to reinterpret the existing mesh data structures. We evaluate the scaling performance of the modified version of ParFlow, demonstrating good weak and strong scaling up to 458k cores of the Juqueen supercomputer, and test an example application at large scale.Comment: The final publication is available at link.springer.co

Improvements in the Scalability of the NASA Goddard Multiscale Modeling Framework for Hurricane Climate Studies

Improving our understanding of hurricane inter-annual variability and the impact of climate change (e.g., doubling CO2 and/or global warming) on hurricanes brings both scientific and computational challenges to researchers. As hurricane dynamics involves multiscale interactions among synoptic-scale flows, mesoscale vortices, and small-scale cloud motions, an ideal numerical model suitable for hurricane studies should demonstrate its capabilities in simulating these interactions. The newly-developed multiscale modeling framework (MMF, Tao et al., 2007) and the substantial computing power by the NASA Columbia supercomputer show promise in pursuing the related studies, as the MMF inherits the advantages of two NASA state-of-the-art modeling components: the GEOS4/fvGCM and 2D GCEs. This article focuses on the computational issues and proposes a revised methodology to improve the MMF's performance and scalability. It is shown that this prototype implementation enables 12-fold performance improvements with 364 CPUs, thereby making it more feasible to study hurricane climate

Multigrid preconditioners for the mixed finite element dynamical core of the LFRic atmospheric model

Due to the wide separation of time scales in geophysical fluid dynamics, semi-implicit time integrators are commonly used in operational atmospheric forecast models. They guarantee the stable treatment of fast (acoustic and gravity) waves, while not suffering from severe restrictions on the timestep size. To propagate the state of the atmosphere forward in time, a non-linear equation for the prognostic variables has to be solved at every timestep. Since the nonlinearity is typically weak, this is done with a small number of Newton- or Picard- iterations, which in turn require the efficient solution of a large system on linear equations with O(106 − 109) unknowns. This linear solve is often the computationally most costly part of the model. In this paper an efficient linear solver for the LFRic next-generation model, currently developed by the Met Office, is described. The model uses an advanced mimetic finite element discretisation which makes the construction of efficient solvers challenging compared to models using standard finite-difference and finite-volume methods. The linear solver hinges on a bespoke multigrid preconditioner of the Schur-complement system for the pressure correction. By comparing to Krylov-subspace methods, the superior performance and robustness of the multigrid algorithm is demonstrated for standard test cases and realistic model setups. In production mode, the model will have to run in parallel on 100,000s of processing elements. As confirmed by numerical experiments, one particular advantage of the multigrid solver is its excellent parallel scalability due to avoiding expensive global reduction operations

The ESCAPE project: Energy-efficient Scalable Algorithms for Weather Prediction at Exascale

Abstract. In the simulation of complex multi-scale flows arising in weather and climate modelling, one of the biggest challenges is to satisfy strict service requirements in terms of time to solution and to satisfy budgetary constraints in terms of energy to solution, without compromising the accuracy and stability of the application. These simulations require algorithms that minimise the energy footprint along with the time required to produce a solution, maintain the physically required level of accuracy, are numerically stable, and are resilient in case of hardware failure. The European Centre for Medium-Range Weather Forecasts (ECMWF) led the ESCAPE (Energy-efficient Scalable Algorithms for Weather Prediction at Exascale) project, funded by Horizon 2020 (H2020) under the FET-HPC (Future and Emerging Technologies in High Performance Computing) initiative. The goal of ESCAPE was to develop a sustainable strategy to evolve weather and climate prediction models to next-generation computing technologies. The project partners incorporate the expertise of leading European regional forecasting consortia, university research, experienced high-performance computing centres, and hardware vendors. This paper presents an overview of the ESCAPE strategy: (i) identify domain-specific key algorithmic motifs in weather prediction and climate models (which we term Weather & Climate Dwarfs), (ii) categorise them in terms of computational and communication patterns while (iii) adapting them to different hardware architectures with alternative programming models, (iv) analyse the challenges in optimising, and (v) find alternative algorithms for the same scheme. The participating weather prediction models are the following: IFS (Integrated Forecasting System); ALARO, a combination of AROME (Application de la Recherche à l'Opérationnel à Meso-Echelle) and ALADIN (Aire Limitée Adaptation Dynamique Développement International); and COSMO–EULAG, a combination of COSMO (Consortium for Small-scale Modeling) and EULAG (Eulerian and semi-Lagrangian fluid solver). For many of the weather and climate dwarfs ESCAPE provides prototype implementations on different hardware architectures (mainly Intel Skylake CPUs, NVIDIA GPUs, Intel Xeon Phi, Optalysys optical processor) with different programming models. The spectral transform dwarf represents a detailed example of the co-design cycle of an ESCAPE dwarf. The dwarf concept has proven to be extremely useful for the rapid prototyping of alternative algorithms and their interaction with hardware; e.g. the use of a domain-specific language (DSL). Manual adaptations have led to substantial accelerations of key algorithms in numerical weather prediction (NWP) but are not a general recipe for the performance portability of complex NWP models. Existing DSLs are found to require further evolution but are promising tools for achieving the latter. Measurements of energy and time to solution suggest that a future focus needs to be on exploiting the simultaneous use of all available resources in hybrid CPU–GPU arrangements

A Review of Element-Based Galerkin Methods for Numerical Weather Prediction: Finite Elements, Spectral Elements, and Discontinuous Galerkin

Numerical weather prediction (NWP) is in a period of transition. As resolutions increase, global models are moving towards fully nonhydrostatic dynamical cores, with the local and global models using the same governing equations; therefore we have reached a point where it will be necessary to use a single model for both applications. The new dynamical cores at the heart of these unified models are designed to scale efficiently on clusters with hundreds of thousands or even millions of CPU cores and GPUs. Operational and research NWP codes currently use a wide range of numerical methods: finite differences, spectral transform, finite volumes and, increasingly, finite/spectral elements and discontinuous Galerkin, which constitute element-based Galerkin (EBG) methods.Due to their important role in this transition, will EBGs be the dominant power behind NWP in the next 10 years, or will they just be one of many methods to choose from? One decade after the review of numerical methods for atmospheric modeling by Steppeler et al. (Meteorol Atmos Phys 82:287–301, 2003), this review discusses EBG methods as a viable numerical approach for the next-generation NWP models. One well-known weakness of EBG methods is the generation of unphysical oscillations in advection-dominated flows; special attention is hence devoted to dissipation-based stabilization methods. Since EBGs are geometrically flexible and allow both conforming and non-conforming meshes, as well as grid adaptivity, this review is concluded with a short overview of how mesh generation and dynamic mesh refinement are becoming as important for atmospheric modeling as they have been for engineering applications for many years.The authors would like to thank Prof. Eugenio Oñate (U. Politècnica de Catalunya) for his invitation to submit this review article. They are also thankful to Prof. Dale Durran (U. Washington), Dr. Tommaso Benacchio (Met Office), and Dr. Matias Avila (BSC-CNS) for their comments and corrections, as well as insightful discussion with Sam Watson, Consulting Software Engineer (Exa Corp.) Most of the contribution to this article by the first author stems from his Ph.D. thesis carried out at the Barcelona Supercomputing Center (BSCCNS) and Universitat Politècnica de Catalunya, Spain, supported by a BSC-CNS student grant, by Iberdrola Energías Renovables, and by grant N62909-09-1-4083 of the Office of Naval Research Global. At NPS, SM, AM, MK, and FXG were supported by the Office of Naval Research through program element PE-0602435N, the Air Force Office of Scientific Research through the Computational Mathematics program, and the National Science Foundation (Division of Mathematical Sciences) through program element 121670. The scalability studies of the atmospheric model NUMA that are presented in this paper used resources of the Argonne Leadership Computing Facility, which is a DOE Office of Science User Facility supported under Contract DE-AC02-06CH11357. SM, MK, and AM are grateful to the National Research Council of the National Academies.Peer ReviewedPostprint (author's final draft

Variational multiscale stabilization of finite and spectral elements for dry and moist atmospheric problems

In this thesis the finite and spectral element methods (FEM and SEM, respectively) applied to problems in atmospheric simulations are explored through the common thread of Variational Multiscale Stabilization (VMS). This effort is justified by three main reasons. (i) the recognized need for new solvers that can efficiently execute on massively parallel architectures ¿a spreading framework in most fields of computational physics in which numerical weather prediction (NWP) occupies a prominent position. Element-based methods (e.g. FEM, SEM, discontinuous Galerkin) have important advantages in parallel code development; (ii) the inherent flexibility of these methods with respect to the geometry of the grid makes them a great candidate for dynamically adaptive atmospheric codes; and (iii) the localized diffusion provided by VMS represents an improvement in the accurate solution of multi-physics problems where artificial diffusion may fail. Its application to atmospheric simulations is a novel approach within a field of research that is still open. First, FEM and VMS are described and derived for the solution of stratified low Mach number flows in the context of dry atmospheric dynamics. The validity of the method to simulate stratified flows is assessed using standard two- and three-dimensional benchmarks accepted by NWP practitioners. The problems include thermal and gravity driven simulations. It will be shown that stability is retained in the regimes of interest and a numerical comparison against results from the the literature will be discussed. Second, the ability of VMS to stabilize the FEM solution of advection-dominated problems (i.e. Euler and transport equations) is taken further by the implementation of VMS as a stabilizing tool for high-order spectral elements with advection-diffusion problems. To the author¿s knowledge, this is an original contribution to the literature of high order spectral elements involved with transport in the atmosphere. The problem of monotonicity-preserving high order methods is addressed by combining VMS-stabilized SEM with a discontinuity capturing technique. This is an alternative to classical filters to treat the Gibbs oscillations that characterize high-order schemes. To conclude, a microphysics scheme is implemented within the finite element Euler solver, as a first step toward realistic atmospheric simulations. Kessler microphysics is used to simulate the formation of warm, precipitating clouds. This last part combines the solution of the Euler equations for stratified flows with the solution of a system of transport equations for three classes of water: water vapor, cloud water, and rain. The method is verified using idealized two- and three-dimensional storm simulations.En esta tesis los métodos de elementos finitos y espectrales (FEM - finite element method y SEM- spectral element method, respectivamente), aplicados a los problemas de simulaciones atmosféricas, se exploran a través del método de estabilización conocidocomo Variational Multiscale Stabilization (VMS). Tres razones fundamentales justifican este esfuerzo: (i) la necesidad de tener nuevos métodos de solución de las ecuaciones diferenciales a las derivadas parciales usando máquinas paralelas de gran escala –un entorno en expansión en muchos campos de la mecánica computacional, dentro de la cual la predicción numérica de la dinámica atmosférica (NWP-numerical weather prediction)representa una aplicación importante. Métodos del tipo basado en elementos(por ejemplo, FEM, SEM, Galerkin discontinuo) presentan grandes ventajas en el desarrollo de códigos paralelos; (ii) la flexibilidad intrínseca de tales métodos respecto a lageometría de la malla computacional hace que esos métodos sean los candidatos ideales para códigos atmosféricos con mallas adaptativas; y (iii) la difusión localizada que VMSintroduce representa una mejora en las soluciones de problemas con física compleja en los cuales la difusión artificial clásica no funcionaría. La aplicación de FEM o SEM con VMS a problemas de simulaciones atmosféricas es una estrategia innovadora en un campo de investigación abierto. En primera instancia, FEM y VMS vienen descritos y derivados para la solución de flujos estratificados a bajo número de Mach en el contexto de la dinámica atmosférica. La validez del método para simular flujos estratificados es verificada por medio de test estándar aceptado por la comunidad dentro del campo deNWP. Los test incluyen simulaciones de flujos térmicos con efectos de gravedad. Se demostrará que la estabilidad del método numérico se preserva dentro de los regímenesde interés y se discutirá una comparación numérica de los resultados frente a aquellos hallados en la literatura. En segunda instancia, la capacidad de VMS para estabilizarmétodos FEM en problemas de advección dominante (i.e. ecuaciones de Euler y ecuaciones de transporte) se implementa además en la solución a elementos espectrales de alto orden en problemas de advección-difusión. Hasta donde el autor sabe, esta es una contribución original a la literatura de métodos basados en elementos espectrales en problemas de transporte atmosférico. El problema de monotonicidad con métodos de alto orden es tratado mediante la combinación de SEM+VMS con una técnica de shockcapturing para un mejor tratamiento de las discontinuidades. Esta es una alternativa a los filtros que normalmente se aplican a SEM para eilminar las oscilaciones de Gibbsque caracterizan las soluciones de alto orden. Como último punto, se implementa un esquema de humedad acoplado con el núcleo en elementos finitos; este es un primer paso hacia simulaciones atmosféricas más realistas. La microfísica de Kessler se emplea para simular la formación de nubes y tormentas cálidas (warm clouds: no permite la formación de hielo). Esta última parte combina la solución de las ecuaciones de Eulerpara atmósferas estratificadas con la solución de un sistema de ecuaciones de transporte de tres estados de agua: vapor, nubes y lluvia. La calidad del método es verificadautilizando simulaciones de tormenta en dos y tres dimensiones