Optimizing an MPI weather forecasting model via processor virtualization

Abstract—Weather forecasting models are computationally intensive applications. These models are typically executed in parallel machines and a major obstacle for their scalability is load imbalance. The causes of such imbalance are either static (e.g. topography) or dynamic (e.g. shortwave radiation, moving thunderstorms). Various techniques, often embedded in the application’s source code, have been used to address both sources. However, these techniques are inflexible and hard to use in legacy codes. In this paper, we demonstrate the effectiveness of processor virtualization for dynamically balancing the load in BRAMS, a mesoscale weather forecasting model based on MPI paral-lelization. We use the Charm++ infrastructure, with its over-decomposition and object-migration capabilities, to move sub-domains across processors during execution of the model. Pro-cessor virtualization enables better overlap between computation and communication and improved cache efficiency. Furthermore, by employing an appropriate load balancer, we achieve better processor utilization while requiring minimal changes to the model’s code. I

The ICON Earth System Model Version 1.0

This work documents ICON-ESM 1.0, the first version of a coupled model based 19 on the ICON framework 20 • Performance of ICON-ESM is assessed by means of CMIP6 DECK experiments 21 at standard CMIP-type resolution 22 • ICON-ESM reproduces the observed temperature evolution. Biases in clouds, winds, 23 sea-ice, and ocean properties are larger than in MPI-ESM. Abstract 25 This work documents the ICON-Earth System Model (ICON-ESM V1.0), the first cou-26 pled model based on the ICON (ICOsahedral Non-hydrostatic) framework with its un-27 structured, icosahedral grid concept. The ICON-A atmosphere uses a nonhydrostatic dy-28 namical core and the ocean model ICON-O builds on the same ICON infrastructure, but 29 applies the Boussinesq and hydrostatic approximation and includes a sea-ice model. The 30 ICON-Land module provides a new framework for the modelling of land processes and 31 the terrestrial carbon cycle. The oceanic carbon cycle and biogeochemistry are repre-32 sented by the Hamburg Ocean Carbon Cycle module. We describe the tuning and spin-33 up of a base-line version at a resolution typical for models participating in the Coupled 34 Model Intercomparison Project (CMIP). The performance of ICON-ESM is assessed by 35 means of a set of standard CMIP6 simulations. Achievements are well-balanced top-of-36 atmosphere radiation, stable key climate quantities in the control simulation, and a good 37 representation of the historical surface temperature evolution. The model has overall bi-38 ases, which are comparable to those of other CMIP models, but ICON-ESM performs 39 less well than its predecessor, the Max Planck Institute Earth System Model. Problem-40 atic biases are diagnosed in ICON-ESM in the vertical cloud distribution and the mean 41 zonal wind field. In the ocean, sub-surface temperature and salinity biases are of con-42 cern as is a too strong seasonal cycle of the sea-ice cover in both hemispheres. ICON-43 ESM V1.0 serves as a basis for further developments that will take advantage of ICON-44 specific properties such as spatially varying resolution, and configurations at very high 45 resolution. 46 Plain Language Summary 47 ICON-ESM is a completely new coupled climate and earth system model that ap-48 plies novel design principles and numerical techniques. The atmosphere model applies 49 a non-hydrostatic dynamical core, both atmosphere and ocean models apply unstruc-50 tured meshes, and the model is adapted for high-performance computing systems. This 51 article describes how the component models for atmosphere, land, and ocean are cou-52 pled together and how we achieve a stable climate by setting certain tuning parameters 53 and performing sensitivity experiments. We evaluate the performance of our new model 54 by running a set of experiments under pre-industrial and historical climate conditions 55 as well as a set of idealized greenhouse-gas-increase experiments. These experiments were 56 designed by the Coupled Model Intercomparison Project (CMIP) and allow us to com-57 pare the results to those from other CMIP models and the predecessor of our model, the 58 Max Planck Institute for Meteorology Earth System Model. While we diagnose overall 59 satisfactory performance, we find that ICON-ESM features somewhat larger biases in 60 several quantities compared to its predecessor at comparable grid resolution. We empha-61 size that the present configuration serves as a basis from where future development steps 62 will open up new perspectives in earth system modellin

The COSMO-CLM 4.8 regional climate model coupled to regional ocean, land surface and global earth system models using OASIS3-MCT: description and performance

We developed a coupled regional climate system model based on the CCLM regional climate model. Within this model system, using OASIS3-MCT as a coupler, CCLM can be coupled to two land surface models (the Community Land Model (CLM) and VEG3D), the NEMO-MED12 regional ocean model for the Mediterranean Sea, two ocean models for the North and Baltic seas (NEMO-NORDIC and TRIMNP+CICE) and the MPI-ESM Earth system model. We first present the different model components and the unified OASIS3-MCT interface which handles all couplings in a consistent way, minimising the model source code modifications and defining the physical and numerical aspects of the couplings. We also address specific coupling issues like the handling of different domains, multiple usage of the MCT library and exchange of 3-D fields. We analyse and compare the computational performance of the different couplings based on real-case simulations over Europe. The usage of the LUCIA tool implemented in OASIS3-MCT enables the quantification of the contributions of the coupled components to the overall coupling cost. These individual contributions are (1) cost of the model(s) coupled, (2) direct cost of coupling including horizontal interpolation and communication between the components, (3) load imbalance, (4) cost of different usage of processors by CCLM in coupled and stand-alone mode and (5) residual cost including i.a. CCLM additional computations. Finally a procedure for finding an optimum processor configuration for each of the couplings was developed considering the time to solution, computing cost and parallel efficiency of the simulation. The optimum configurations are presented for sequential, concurrent and mixed (sequential+concurrent) coupling layouts. The procedure applied can be regarded as independent of the specific coupling layout and coupling details. We found that the direct cost of coupling, i.e. communications and horizontal interpolation, in OASIS3-MCT remains below 7 % of the CCLM stand-alone cost for all couplings investigated. This is in particular true for the exchange of 450 2-D fields between CCLM and MPI-ESM. We identified remaining limitations in the coupling strategies and discuss possible future improvements of the computational efficiency

NAS technical summaries. Numerical aerodynamic simulation program, March 1992 - February 1993

NASA created the Numerical Aerodynamic Simulation (NAS) Program in 1987 to focus resources on solving critical problems in aeroscience and related disciplines by utilizing the power of the most advanced supercomputers available. The NAS Program provides scientists with the necessary computing power to solve today's most demanding computational fluid dynamics problems and serves as a pathfinder in integrating leading-edge supercomputing technologies, thus benefitting other supercomputer centers in government and industry. The 1992-93 operational year concluded with 399 high-speed processor projects and 91 parallel projects representing NASA, the Department of Defense, other government agencies, private industry, and universities. This document provides a glimpse at some of the significant scientific results for the year

On the predictability of exceptional error events in wind power forecasting —an ultra large ensemble approach—

Exceptional error events in wind power forecasting impose a major obstacle to today’s reliable power supply. The predictability of such error events is fundamentally restricted by the underlying weather forecast, resting on limitations of state-of-the-art numerical prediction systems. This work aims to identify such imminent forecast errors applying a probabilistic approach. To this end, the standard sizes of meteorological ensembles are increased from O(10) to an ultra large ensemble size of O(1000) members to accomplish an improved approximation of the probability density function. For this purpose, a novel approach of an ensemble control system named ESIAS-met has been developed on a Petaflop architecture. Further, an increased ensemble size favors the application of nonlinear data assimilation techniques based on the particle filter, while imposing the challenge of growing computational expenses of a resampling step within the particle filter algorithm. ESIAS-met presents a computationally efficient solution to the problem by realizing a parallel execution of the ensemble. Performance measurements demonstrate strong scalability of the system with up to 4096 members. Moreover, the computational expenses of a particle filter resampling step are shown to become independent of the ensemble size. The ESIAS-met system is further applied to investigate the benefit of an increased ensemble size on the predictability of recent exceptional error events. The analysis reveals, that despite the large ensemble size, the forecast error is only represented by single outliers. Higher order moments prove to provide a robust measure of the proper direction of forecast error and assess their likelihood of appearance. It is shown, that at least O(100) ensemble members are needed to resolve the higher order moments sufficiently well. Hence, the results achieved in this work yield important potential for future warning capabilities of exceptional error events

The WWRP Polar Prediction Project (PPP)

Mission statement: “Promote cooperative international research enabling development of improved weather and environmental prediction services for the polar regions, on time scales from hours to seasonal”. Increased economic, transportation and research activities in polar regions are leading to more demands for sustained and improved availability of predictive weather and climate information to support decision-making. However, partly as a result of a strong emphasis of previous international efforts on lower and middle latitudes, many gaps in weather, sub-seasonal and seasonal forecasting in polar regions hamper reliable decision making in the Arctic, Antarctic and possibly the middle latitudes as well. In order to advance polar prediction capabilities, the WWRP Polar Prediction Project (PPP) has been established as one of three THORPEX (THe Observing System Research and Predictability EXperiment) legacy activities. The aim of PPP, a ten year endeavour (2013-2022), is to promote cooperative international research enabling development of improved weather and environmental prediction services for the polar regions, on hourly to seasonal time scales. In order to achieve its goals, PPP will enhance international and interdisciplinary collaboration through the development of strong linkages with related initiatives; strengthen linkages between academia, research institutions and operational forecasting centres; promote interactions and communication between research and stakeholders; and foster education and outreach. Flagship research activities of PPP include sea ice prediction, polar-lower latitude linkages and the Year of Polar Prediction (YOPP) - an intensive observational, coupled modelling, service-oriented research and educational effort in the period mid-2017 to mid-2019

The GEOS-5 Data Assimilation System-Documentation of Versions 5.0.1, 5.1.0, and 5.2.0

This report documents the GEOS-5 global atmospheric model and data assimilation system (DAS), including the versions 5.0.1, 5.1.0, and 5.2.0, which have been implemented in products distributed for use by various NASA instrument team algorithms and ultimately for the Modem Era Retrospective analysis for Research and Applications (MERRA). The DAS is the integration of the GEOS-5 atmospheric model with the Gridpoint Statistical Interpolation (GSI) Analysis, a joint analysis system developed by the NOAA/National Centers for Environmental Prediction and the NASA/Global Modeling and Assimilation Office. The primary performance drivers for the GEOS DAS are temperature and moisture fields suitable for the EOS instrument teams, wind fields for the transport studies of the stratospheric and tropospheric chemistry communities, and climate-quality analyses to support studies of the hydrological cycle through MERRA. The GEOS-5 atmospheric model has been approved for open source release and is available from: http://opensource.gsfc.nasa.gov/projects/GEOS-5/GEOS-5.php

Numerical evaluation of aerodynamic roughness of the built environment and complex terrain

Aerodynamic drag in the atmospheric boundary layer (ABL) is affected by the structure and density of obstacles (surface roughness) and nature of the terrain (topography). In building codes and standards, average roughness is usually determined somewhat subjectively by examination of aerial photographs. For detailed wind mapping, boundary layer wind tunnel (BLWT) testing is usually recommended. This may not be cost effective for many projects, in which case numerical studies become good alternatives. This thesis examines Computational Fluid Dynamics (CFD) for evaluation of aerodynamic roughness of the built environment and complex terrain. The present study started from development of an in-house CFD software tailored for ABL simulations. A three-dimensional finite-volume code was developed using flexible polyhedral elements as building blocks. The program is parallelized using MPI to run on clusters of processors so that micro-scale simulations can be conducted quickly. The program can also utilize the power of latest technology in high performance computing, namely GPUs. Various turbulence models including mixing-length, RANS, and LES models are implemented, and their suitability for ABL simulations assessed. Then the effect of surface roughness alone on wind profiles is assessed using CFD. Cases with various levels of complexity are considered including simplified models with roughness blocks of different arrangement, multiple roughness patches, semi-idealized urban model, and real built environment. Comparison with BLWT data for the first three cases showed good agreement thereby justifying explicit three-dimensional numerical approach. Due to lack of validation data, the real built environment case served only to demonstrate use of CFD for such purposes. Finally, the effect of topographic features on wind profiles was investigated using CFD. This work extends prior work done by the research team on multiple idealized two-dimensional topographic features to more elaborate three-dimensional simulations. It is found that two-dimensional simulations overestimate speed up over crests of hills and also show larger recirculation zones. The current study also emphasized turbulence characterization behind hills. Finally a real complex terrain case of the well-known Askervein hill was simulated and the results validated against published field observations. In general the results obtained from the current simulations compared well with those reported in literature