Multigrid preconditioners for the mixed finite element dynamical core of the LFRic atmospheric model

Due to the wide separation of time scales in geophysical fluid dynamics, semi-implicit time integrators are commonly used in operational atmospheric forecast models. They guarantee the stable treatment of fast (acoustic and gravity) waves, while not suffering from severe restrictions on the timestep size. To propagate the state of the atmosphere forward in time, a non-linear equation for the prognostic variables has to be solved at every timestep. Since the nonlinearity is typically weak, this is done with a small number of Newton- or Picard- iterations, which in turn require the efficient solution of a large system on linear equations with O(106 − 109) unknowns. This linear solve is often the computationally most costly part of the model. In this paper an efficient linear solver for the LFRic next-generation model, currently developed by the Met Office, is described. The model uses an advanced mimetic finite element discretisation which makes the construction of efficient solvers challenging compared to models using standard finite-difference and finite-volume methods. The linear solver hinges on a bespoke multigrid preconditioner of the Schur-complement system for the pressure correction. By comparing to Krylov-subspace methods, the superior performance and robustness of the multigrid algorithm is demonstrated for standard test cases and realistic model setups. In production mode, the model will have to run in parallel on 100,000s of processing elements. As confirmed by numerical experiments, one particular advantage of the multigrid solver is its excellent parallel scalability due to avoiding expensive global reduction operations

HOMMEXX 1.0: a performance-portable atmospheric dynamical core for the Energy Exascale Earth System Model

We present an architecture-portable and performant implementation of the atmospheric dynamical core (High-Order Methods Modeling Environment, HOMME) of the Energy Exascale Earth System Model (E3SM). The original Fortran implementation is highly performant and scalable on conventional architectures using the Message Passing Interface (MPI) and Open MultiProcessor (OpenMP) programming models. We rewrite the model in C++ and use the Kokkos library to express on-node parallelism in a largely architecture-independent implementation. Kokkos provides an abstraction of a compute node or device, layout-polymorphic multidimensional arrays, and parallel execution constructs. The new implementation achieves the same or better performance on conventional multicore computers and is portable to GPUs. We present performance data for the original and new implementations on multiple platforms, on up to 5400 compute nodes, and study several aspects of the single- and multi-node performance characteristics of the new implementation on conventional CPU (e.g., Intel Xeon), many core CPU (e.g., Intel Xeon Phi Knights Landing), and Nvidia V100 GPU.</p

Das unstetige Galerkinverfahren für Strömungen mit freier Oberfläche und im Grundwasserbereich in geophysikalischen Anwendungen

Free surface flows and subsurface flows appear in a broad range of geophysical applications and in many environmental settings situations arise which even require the coupling of free surface and subsurface flows. Many of these application scenarios are characterized by large domain sizes and long simulation times. Hence, they need considerable amounts of computational work to achieve accurate solutions and the use of efficient algorithms and high performance computing resources to obtain results within a reasonable time frame is mandatory. Discontinuous Galerkin methods are a class of numerical methods for solving differential equations that share characteristics with methods from the finite volume and finite element frameworks. They feature high approximation orders, offer a large degree of flexibility, and are well-suited for parallel computing. This thesis consists of eight articles and an extended summary that describe the application of discontinuous Galerkin methods to mathematical models including free surface and subsurface flow scenarios with a strong focus on computational aspects. It covers discretization and implementation aspects, the parallelization of the method, and discrete stability analysis of the coupled model.Für viele geophysikalische Anwendungen spielen Strömungen mit freier Oberfläche und im Grundwasserbereich oder sogar die Kopplung dieser beiden eine zentrale Rolle. Oftmals charakteristisch für diese Anwendungsszenarien sind große Rechengebiete und lange Simulationszeiten. Folglich ist das Berechnen akkurater Lösungen mit beträchtlichem Rechenaufwand verbunden und der Einsatz effizienter Lösungsverfahren sowie von Techniken des Hochleistungsrechnens obligatorisch, um Ergebnisse innerhalb eines annehmbaren Zeitrahmens zu erhalten. Unstetige Galerkinverfahren stellen eine Gruppe numerischer Verfahren zum Lösen von Differentialgleichungen dar, und kombinieren Eigenschaften von Methoden der Finiten Volumen- und Finiten Elementeverfahren. Sie ermöglichen hohe Approximationsordnungen, bieten einen hohen Grad an Flexibilität und sind für paralleles Rechnen gut geeignet. Diese Dissertation besteht aus acht Artikeln und einer erweiterten Zusammenfassung, in diesen die Anwendung unstetiger Galerkinverfahren auf mathematische Modelle inklusive solcher für Strömungen mit freier Oberfläche und im Grundwasserbereich beschrieben wird. Die behandelten Themen umfassen Diskretisierungs- und Implementierungsaspekte, die Parallelisierung der Methode sowie eine diskrete Stabilitätsanalyse des gekoppelten Modells

Non-modal analysis of spectral element methods: Towards accurate and robust large-eddy simulations

High-order spectral element methods (SEM) for large-eddy simulation (LES) are still very limited in industry. One of the main reasons behind this is the lack of robustness of SEM for under-resolved simulations, which can lead to the failure of the computation or to inaccurate results, aspects that are critical in an industrial setting. To help address this issue, we introduce a non-modal analysis technique that characterizes the numerical diffusion properties of spectral element methods for linear convection–diffusion problems, including the scales affected by numerical diffusion and the relationship between the amount of numerical diffusion and the level of under-resolution in the simulation. This framework differs from traditional eigenanalysis techniques in that all eigenmodes are taken into account with no need to differentiate them as physical or unphysical. While strictly speaking only valid for linear problems, the non-modal analysis is devised so that it can give critical insights for under-resolved nonlinear problems. For example, why do SEM sometimes suffer from numerical stability issues in LES? And, why do they at other times be robust and successfully predict under-resolved turbulent flows even without a subgrid-scale model? The answer to these questions in turn provides crucial guidelines to construct more robust and accurate schemes for LES. For illustration purposes, the non-modal analysis is applied to the hybridized discontinuous Galerkin methods as representatives of SEM. The effects of the polynomial order, the upwinding parameter and the Péclet number on the so-called short-term diffusion of the scheme are investigated. From a non-modal analysis point of view, and for the particular case of hybridized discontinuous Galerkin methods, polynomial orders between 2 and 4 with standard upwinding are found to be well suited for under-resolved turbulence simulations. For lower polynomial orders, diffusion is introduced in scales that are much larger than the grid resolution. For higher polynomial orders, as well as for strong under/over-upwinding, robustness issues can be expected due to low and non-monotonic numerical diffusion. The non-modal analysis results are tested against under-resolved turbulence simulations of the Burgers, Euler and Navier–Stokes equations. While devised in the linear setting, non-modal analysis successfully predicts the behavior of the scheme in the nonlinear problems considered. Although the focus of this paper is on LES, the non-modal analysis can be applied to other simulation fields characterized by under-resolved scales

MPAS-Albany Land Ice (MALI): a variable-resolution ice sheet model for Earth system modeling using Voronoi grids

We introduce MPAS-Albany Land Ice (MALI) v6.0, a new variable-resolution land ice model that uses unstructured Voronoi grids on a plane or sphere. MALI is built using the Model for Prediction Across Scales (MPAS) framework for developing variable-resolution Earth system model components and the Albany multi-physics code base for the solution of coupled systems of partial differential equations, which itself makes use of Trilinos solver libraries. MALI includes a three-dimensional first-order momentum balance solver (Blatter–Pattyn) by linking to the Albany-LI ice sheet velocity solver and an explicit shallow ice velocity solver. The evolution of ice geometry and tracers is handled through an explicit first-order horizontal advection scheme with vertical remapping. The evolution of ice temperature is treated using operator splitting of vertical diffusion and horizontal advection and can be configured to use either a temperature or enthalpy formulation. MALI includes a mass-conserving subglacial hydrology model that supports distributed and/or channelized drainage and can optionally be coupled to ice dynamics. Options for calving include eigencalving, which assumes that the calving rate is proportional to extensional strain rates. MALI is evaluated against commonly used exact solutions and community benchmark experiments and shows the expected accuracy. Results for the MISMIP3d benchmark experiments with MALI's Blatter–Pattyn solver fall between published results from Stokes and L1L2 models as expected. We use the model to simulate a semi-realistic Antarctic ice sheet problem following the initMIP protocol and using 2&thinsp;km resolution in marine ice sheet regions. MALI is the glacier component of the Energy Exascale Earth System Model (E3SM) version 1, and we describe current and planned coupling to other E3SM components.</p

Hybridizable compatible finite element discretizations for numerical weather prediction: implementation and analysis

There is a current explosion of interest in new numerical methods for atmospheric modeling. A driving force behind this is the need to be able to simulate, with high efficiency, large-scale geophysical flows on increasingly more parallel computer systems. Many current operational models, including that of the UK Met Office, depend on orthogonal meshes, such as the latitude-longitude grid. This facilitates the development of finite difference discretizations with favorable numerical properties. However, such methods suffer from the ``pole problem," which prohibits the model to make efficient use of a large number of computing processors due to excessive concentration of grid-points at the poles. Recently developed finite element discretizations, known as ``compatible" finite elements, avoid this issue while maintaining the key numerical properties essential for accurate geophysical simulations. Moreover, these properties can be obtained on arbitrary, non-orthogonal meshes. However, the efficient solution of the resulting discrete systems depend on transforming the mixed velocity-pressure (or velocity-pressure-buoyancy) system into an elliptic problem for the pressure. This is not so straightforward within the compatible finite element framework due to inter-element coupling. This thesis supports the proposition that systems arising from compatible finite element discretizations can be solved efficiently using a technique known as ``hybridization." Hybridization removes inter-element coupling while maintaining the desired numerical properties. This permits the construction of sparse, elliptic problems, for which fast solver algorithms are known, using localized algebra. We first introduce the technique for compatible finite element discretizations of simplified atmospheric models. We then develop a general software abstraction for the rapid implementation and composition of hybridization methods, with an emphasis on preconditioning. Finally, we extend the technique for a new compatible method for the full, compressible atmospheric equations used in operational models.Open Acces