The ESCAPE project : Energy-efficient Scalable Algorithms for Weather Prediction at Exascale

In the simulation of complex multi-scale flows arising in weather and climate modelling, one of the biggest challenges is to satisfy strict service requirements in terms of time to solution and to satisfy budgetary constraints in terms of energy to solution, without compromising the accuracy and stability of the application. These simulations require algorithms that minimise the energy footprint along with the time required to produce a solution, maintain the physically required level of accuracy, are numerically stable, and are resilient in case of hardware failure. The European Centre for Medium-Range Weather Forecasts (ECMWF) led the ESCAPE (Energy-efficient Scalable Algorithms for Weather Prediction at Exascale) project, funded by Horizon 2020 (H2020) under the FET-HPC (Future and Emerging Technologies in High Performance Computing) initiative. The goal of ESCAPE was to develop a sustainable strategy to evolve weather and climate prediction models to next-generation computing technologies. The project partners incorporate the expertise of leading European regional forecasting consortia, university research, experienced high-performance computing centres, and hardware vendors. This paper presents an overview of the ESCAPE strategy: (i) identify domain-specific key algorithmic motifs in weather prediction and climate models (which we term Weather & Climate Dwarfs), (ii) categorise them in terms of computational and communication patterns while (iii) adapting them to different hardware architectures with alternative programming models, (iv) analyse the challenges in optimising, and (v) find alternative algorithms for the same scheme. The participating weather prediction models are the following: IFS (Integrated Forecasting System); ALARO, a combination of AROME (Application de la Recherche a l'Operationnel a Meso-Echelle) and ALADIN (Aire Limitee Adaptation Dynamique Developpement International); and COSMO-EULAG, a combination of COSMO (Consortium for Small-scale Modeling) and EULAG (Eulerian and semi-Lagrangian fluid solver). For many of the weather and climate dwarfs ESCAPE provides prototype implementations on different hardware architectures (mainly Intel Skylake CPUs, NVIDIA GPUs, Intel Xeon Phi, Optalysys optical processor) with different programming models. The spectral transform dwarf represents a detailed example of the co-design cycle of an ESCAPE dwarf. The dwarf concept has proven to be extremely useful for the rapid prototyping of alternative algorithms and their interaction with hardware; e.g. the use of a domain-specific language (DSL). Manual adaptations have led to substantial accelerations of key algorithms in numerical weather prediction (NWP) but are not a general recipe for the performance portability of complex NWP models. Existing DSLs are found to require further evolution but are promising tools for achieving the latter. Measurements of energy and time to solution suggest that a future focus needs to be on exploiting the simultaneous use of all available resources in hybrid CPU-GPU arrangements

Efficient Multigrid Preconditioners for Atmospheric Flow Simulations at High Aspect Ratio

Many problems in fluid modelling require the efficient solution of highly anisotropic elliptic partial differential equations (PDEs) in "flat" domains. For example, in numerical weather- and climate-prediction an elliptic PDE for the pressure correction has to be solved at every time step in a thin spherical shell representing the global atmosphere. This elliptic solve can be one of the computationally most demanding components in semi-implicit semi-Lagrangian time stepping methods which are very popular as they allow for larger model time steps and better overall performance. With increasing model resolution, algorithmically efficient and scalable algorithms are essential to run the code under tight operational time constraints. We discuss the theory and practical application of bespoke geometric multigrid preconditioners for equations of this type. The algorithms deal with the strong anisotropy in the vertical direction by using the tensor-product approach originally analysed by B\"{o}rm and Hiptmair [Numer. Algorithms, 26/3 (2001), pp. 219-234]. We extend the analysis to three dimensions under slightly weakened assumptions, and numerically demonstrate its efficiency for the solution of the elliptic PDE for the global pressure correction in atmospheric forecast models. For this we compare the performance of different multigrid preconditioners on a tensor-product grid with a semi-structured and quasi-uniform horizontal mesh and a one dimensional vertical grid. The code is implemented in the Distributed and Unified Numerics Environment (DUNE), which provides an easy-to-use and scalable environment for algorithms operating on tensor-product grids. Parallel scalability of our solvers on up to 20,480 cores is demonstrated on the HECToR supercomputer.Comment: 22 pages, 6 Figures, 2 Table

SPH-EXA: Enhancing the Scalability of SPH codes Via an Exascale-Ready SPH Mini-App

Numerical simulations of fluids in astrophysics and computational fluid dynamics (CFD) are among the most computationally-demanding calculations, in terms of sustained floating-point operations per second, or FLOP/s. It is expected that these numerical simulations will significantly benefit from the future Exascale computing infrastructures, that will perform 10^18 FLOP/s. The performance of the SPH codes is, in general, adversely impacted by several factors, such as multiple time-stepping, long-range interactions, and/or boundary conditions. In this work an extensive study of three SPH implementations SPHYNX, ChaNGa, and XXX is performed, to gain insights and to expose any limitations and characteristics of the codes. These codes are the starting point of an interdisciplinary co-design project, SPH-EXA, for the development of an Exascale-ready SPH mini-app. We implemented a rotating square patch as a joint test simulation for the three SPH codes and analyzed their performance on a modern HPC system, Piz Daint. The performance profiling and scalability analysis conducted on the three parent codes allowed to expose their performance issues, such as load imbalance, both in MPI and OpenMP. Two-level load balancing has been successfully applied to SPHYNX to overcome its load imbalance. The performance analysis shapes and drives the design of the SPH-EXA mini-app towards the use of efficient parallelization methods, fault-tolerance mechanisms, and load balancing approaches.Comment: arXiv admin note: substantial text overlap with arXiv:1809.0801

Geodynamic contributions to global climatic change

Orbital and rotational variations perturb the latitudinal and seasonal pattern of incident solar radiation, producing major climatic change on time scales of 10(exp 4)-10(exp 6) years. The orbital variations are oblivious to internal structure and processes, but the rotational variations are not. A program of investigation whose objective would be to explore and quantify three aspects of orbital, rotational, and climatic interactions is described. An important premise of this investigation is the synergism between geodynamics and paleoclimate. Better geophysical models of precessional dynamics are needed in order to accurately reconstruct the radiative input to climate models. Some of the paleoclimate proxy records contain information relevant to solid Earth processes, on time scales which are difficult to constrain otherwise. Specific mechanisms which will be addressed include: (1) climatic consequences of deglacial polar motion; and (2) precessional and climatic consequences of glacially induced perturbations in the gravitational oblateness and partial decoupling of the mantle and core. The approach entails constructing theoretical models of the rotational, deformational, radiative, and climatic response of the Earth to known orbital perturbations, and comparing these with extensive records of paleoclimate proxy data. Several of the mechanisms of interest may participate in previously unrecognized feed-back loops in the climate dynamics system. A new algorithm for estimating climatically diagnostic locations and seasons from the paleoclimate time series is proposed

Physics–Dynamics Coupling in weather, climate and Earth system models: Challenges and recent progress

This is the final version. Available from American Meteorological Society via the DOI in this record.Numerical weather, climate, or Earth system models involve the coupling of components. At a broad level, these components can be classified as the resolved fluid dynamics, unresolved fluid dynamical aspects (i.e., those represented by physical parameterizations such as subgrid-scale mixing), and nonfluid dynamical aspects such as radiation and microphysical processes. Typically, each component is developed, at least initially, independently. Once development is mature, the components are coupled to deliver a model of the required complexity. The implementation of the coupling can have a significant impact on the model. As the error associated with each component decreases, the errors introduced by the coupling will eventually dominate. Hence, any improvement in one of the components is unlikely to improve the performance of the overall system. The challenges associated with combining the components to create a coherent model are here termed physics–dynamics coupling. The issue goes beyond the coupling between the parameterizations and the resolved fluid dynamics. This paper highlights recent progress and some of the current challenges. It focuses on three objectives: to illustrate the phenomenology of the coupling problem with references to examples in the literature, to show how the problem can be analyzed, and to create awareness of the issue across the disciplines and specializations. The topics addressed are different ways of advancing full models in time, approaches to understanding the role of the coupling and evaluation of approaches, coupling ocean and atmosphere models, thermodynamic compatibility between model components, and emerging issues such as those that arise as model resolutions increase and/or models use variable resolutions.Natural Environment Research Council (NERC)National Science FoundationDepartment of Energy Office of Biological and Environmental ResearchPacific Northwest National Laboratory (PNNL)DOE Office of Scienc