Multiphysics simulations of collisionless plasmas

Collisionless plasmas, mostly present in astrophysical and space environments, often require a kinetic treatment as given by the Vlasov equation. Unfortunately, the six-dimensional Vlasov equation can only be solved on very small parts of the considered spatial domain. However, in some cases, e.g. magnetic reconnection, it is sufficient to solve the Vlasov equation in a localized domain and solve the remaining domain by appropriate fluid models. In this paper, we describe a hierarchical treatment of collisionless plasmas in the following way. On the finest level of description, the Vlasov equation is solved both for ions and electrons. The next courser description treats electrons with a 10-moment fluid model incorporating a simplified treatment of Landau damping. At the boundary between the electron kinetic and fluid region, the central question is how the fluid moments influence the electron distribution function. On the next coarser level of description the ions are treated by an 10-moment fluid model as well. It may turn out that in some spatial regions far away from the reconnection zone the temperature tensor in the 10-moment description is nearly isotopic. In this case it is even possible to switch to a 5-moment description. This change can be done separately for ions and electrons. To test this multiphysics approach, we apply this full physics-adaptive simulations to the Geospace Environmental Modeling (GEM) challenge of magnetic reconnection.Comment: 13 pages, 5 figure

Distributed-memory large deformation diffeomorphic 3D image registration

We present a parallel distributed-memory algorithm for large deformation diffeomorphic registration of volumetric images that produces large isochoric deformations (locally volume preserving). Image registration is a key technology in medical image analysis. Our algorithm uses a partial differential equation constrained optimal control formulation. Finding the optimal deformation map requires the solution of a highly nonlinear problem that involves pseudo-differential operators, biharmonic operators, and pure advection operators both forward and back- ward in time. A key issue is the time to solution, which poses the demand for efficient optimization methods as well as an effective utilization of high performance computing resources. To address this problem we use a preconditioned, inexact, Gauss-Newton- Krylov solver. Our algorithm integrates several components: a spectral discretization in space, a semi-Lagrangian formulation in time, analytic adjoints, different regularization functionals (including volume-preserving ones), a spectral preconditioner, a highly optimized distributed Fast Fourier Transform, and a cubic interpolation scheme for the semi-Lagrangian time-stepping. We demonstrate the scalability of our algorithm on images with resolution of up to $1024^3$ on the "Maverick" and "Stampede" systems at the Texas Advanced Computing Center (TACC). The critical problem in the medical imaging application domain is strong scaling, that is, solving registration problems of a moderate size of $256^3$---a typical resolution for medical images. We are able to solve the registration problem for images of this size in less than five seconds on 64 x86 nodes of TACC's "Maverick" system.Comment: accepted for publication at SC16 in Salt Lake City, Utah, USA; November 201

Coupled Vlasov and two-fluid codes on GPUs

We present a way to combine Vlasov and two-fluid codes for the simulation of a collisionless plasma in large domains while keeping full information of the velocity distribution in localized areas of interest. This is made possible by solving the full Vlasov equation in one region while the remaining area is treated by a 5-moment two-fluid code. In such a treatment, the main challenge of coupling kinetic and fluid descriptions is the interchange of physically correct boundary conditions between the different plasma models. In contrast to other treatments, we do not rely on any specific form of the distribution function, e.g. a Maxwellian type. Instead, we combine an extrapolation of the distribution function and a correction of the moments based on the fluid data. Thus, throughout the simulation both codes provide the necessary boundary conditions for each other. A speed-up factor of around 20 is achieved by using GPUs for the computationally expensive solution of the Vlasov equation and an overall factor of at least 60 using the coupling strategy combined with the GPU computation. The coupled codes were then tested on the GEM reconnection challenge

Accelerating Eulerian Fluid Simulation With Convolutional Networks

Efficient simulation of the Navier-Stokes equations for fluid flow is a long standing problem in applied mathematics, for which state-of-the-art methods require large compute resources. In this work, we propose a data-driven approach that leverages the approximation power of deep-learning with the precision of standard solvers to obtain fast and highly realistic simulations. Our method solves the incompressible Euler equations using the standard operator splitting method, in which a large sparse linear system with many free parameters must be solved. We use a Convolutional Network with a highly tailored architecture, trained using a novel unsupervised learning framework to solve the linear system. We present real-time 2D and 3D simulations that outperform recently proposed data-driven methods; the obtained results are realistic and show good generalization properties.Comment: Significant revisio

libcloudph++ 0.2: single-moment bulk, double-moment bulk, and particle-based warm-rain microphysics library in C++

This paper introduces a library of algorithms for representing cloud microphysics in numerical models. The library is written in C++, hence the name libcloudph++. In the current release, the library covers three warm-rain schemes: the single- and double-moment bulk schemes, and the particle-based scheme with Monte-Carlo coalescence. The three schemes are intended for modelling frameworks of different dimensionality and complexity ranging from parcel models to multi-dimensional cloud-resolving (e.g. large-eddy) simulations. A two-dimensional prescribed-flow framework is used in example simulations presented in the paper with the aim of highlighting the library features. The libcloudph++ and all its mandatory dependencies are free and open-source software. The Boost.units library is used for zero-overhead dimensional analysis of the code at compile time. The particle-based scheme is implemented using the Thrust library that allows to leverage the power of graphics processing units (GPU), retaining the possibility to compile the unchanged code for execution on single or multiple standard processors (CPUs). The paper includes complete description of the programming interface (API) of the library and a performance analysis including comparison of GPU and CPU setups.Comment: The library description has been updated to the new library API (i.e. v0.1 -> v0.2 update). The key difference is that the model state variables are now mixing ratios as opposed to densities. The particle-based scheme was supplemented with the "particle recycling" process. Numerous editorial corrections were mad

The ESCAPE project : Energy-efficient Scalable Algorithms for Weather Prediction at Exascale

In the simulation of complex multi-scale flows arising in weather and climate modelling, one of the biggest challenges is to satisfy strict service requirements in terms of time to solution and to satisfy budgetary constraints in terms of energy to solution, without compromising the accuracy and stability of the application. These simulations require algorithms that minimise the energy footprint along with the time required to produce a solution, maintain the physically required level of accuracy, are numerically stable, and are resilient in case of hardware failure. The European Centre for Medium-Range Weather Forecasts (ECMWF) led the ESCAPE (Energy-efficient Scalable Algorithms for Weather Prediction at Exascale) project, funded by Horizon 2020 (H2020) under the FET-HPC (Future and Emerging Technologies in High Performance Computing) initiative. The goal of ESCAPE was to develop a sustainable strategy to evolve weather and climate prediction models to next-generation computing technologies. The project partners incorporate the expertise of leading European regional forecasting consortia, university research, experienced high-performance computing centres, and hardware vendors. This paper presents an overview of the ESCAPE strategy: (i) identify domain-specific key algorithmic motifs in weather prediction and climate models (which we term Weather & Climate Dwarfs), (ii) categorise them in terms of computational and communication patterns while (iii) adapting them to different hardware architectures with alternative programming models, (iv) analyse the challenges in optimising, and (v) find alternative algorithms for the same scheme. The participating weather prediction models are the following: IFS (Integrated Forecasting System); ALARO, a combination of AROME (Application de la Recherche a l'Operationnel a Meso-Echelle) and ALADIN (Aire Limitee Adaptation Dynamique Developpement International); and COSMO-EULAG, a combination of COSMO (Consortium for Small-scale Modeling) and EULAG (Eulerian and semi-Lagrangian fluid solver). For many of the weather and climate dwarfs ESCAPE provides prototype implementations on different hardware architectures (mainly Intel Skylake CPUs, NVIDIA GPUs, Intel Xeon Phi, Optalysys optical processor) with different programming models. The spectral transform dwarf represents a detailed example of the co-design cycle of an ESCAPE dwarf. The dwarf concept has proven to be extremely useful for the rapid prototyping of alternative algorithms and their interaction with hardware; e.g. the use of a domain-specific language (DSL). Manual adaptations have led to substantial accelerations of key algorithms in numerical weather prediction (NWP) but are not a general recipe for the performance portability of complex NWP models. Existing DSLs are found to require further evolution but are promising tools for achieving the latter. Measurements of energy and time to solution suggest that a future focus needs to be on exploiting the simultaneous use of all available resources in hybrid CPU-GPU arrangements

Inertial Coupling Method for particles in an incompressible fluctuating fluid

We develop an inertial coupling method for modeling the dynamics of point-like 'blob' particles immersed in an incompressible fluid, generalizing previous work for compressible fluids. The coupling consistently includes excess (positive or negative) inertia of the particles relative to the displaced fluid, and accounts for thermal fluctuations in the fluid momentum equation. The coupling between the fluid and the blob is based on a no-slip constraint equating the particle velocity with the local average of the fluid velocity, and conserves momentum and energy. We demonstrate that the formulation obeys a fluctuation-dissipation balance, owing to the non-dissipative nature of the no-slip coupling. We develop a spatio-temporal discretization that preserves, as best as possible, these properties of the continuum formulation. In the spatial discretization, the local averaging and spreading operations are accomplished using compact kernels commonly used in immersed boundary methods. We find that the special properties of these kernels make the discrete blob a particle with surprisingly physically-consistent volume, mass, and hydrodynamic properties. We develop a second-order semi-implicit temporal integrator that maintains discrete fluctuation-dissipation balance, and is not limited in stability by viscosity. Furthermore, the temporal scheme requires only constant-coefficient Poisson and Helmholtz linear solvers, enabling a very efficient and simple FFT-based implementation on GPUs. We numerically investigate the performance of the method on several standard test problems...Comment: Contains a number of corrections and an additional Figure 7 (and associated discussion) relative to published versio