WRF4SG: A scientific gateway for climate experiment workflows

Trabajo presentado a la European Geosciences Union General Assembly celebrada en Viena del 7 al 12 de abril de 2013.The Weather Research and Forecasting model (WRF) is a community-driven and public domain model widely used by the weather and climate communities. As opposite to other application-oriented models, WRF provides a flexible and computationally-efficient framework which allows solving a variety of problems for different time-scales, from weather forecast to climate change projection. Furthermore, WRF is also widely used as a research tool in modeling physics, dynamics, and data assimilation by the research community. Climate experiment workflows based on Weather Research and Forecasting (WRF) are nowadays among the one of the most cutting-edge applications. These workflows are complex due to both large storage and the huge number of simulations executed. In order to manage that, we have developed a scientific gateway (SG) called WRF for Scientific Gateway (WRF4SG) based on WS-PGRADE/gUSE and WRF4G frameworks to ease achieve WRF users needs (see [1] and [2]). WRF4SG provides services for different use cases that describe the different interactions between WRF users and the WRF4SG interface in order to show how to run a climate experiment. As WS-PGRADE/gUSE uses portlets (see [1]) to interact with users, its portlets will support these use cases. A typical experiment to be carried on by a WRF user will consist on a high-resolution regional re-forecast. These re-forecasts are common experiments used as input data form wind power energy and natural hazards (wind and precipitation fields). In the cases below, the user is able to access to different resources such as Grid due to the fact that WRF needs a huge amount of computing resources in order to generate useful simulations: - Resource configuration and user authentication: The first step is to authenticate on users’ Grid resources by virtual organizations. After login, the user is able to select which virtual organization is going to be used by the experiment. - Data assimilation: In order to assimilate the data sources, the user has to select them browsing through LFC Portlet. - Design Experiment workflow: In order to configure the experiment, the user will define the type of experiment (i.e. re-forecast), and its attributes to simulate. In this case the main attributes are: the field of interest (wind, precipitation, ...), the start and end date simulation and the requirements of the experiment. - Monitor workflow: In order to monitor the experiment the user will receive notification messages based on events and also the gateway will display the progress of the experiment. - Data storage: Like Data assimilation case, the user is able to browse and view the output data simulations using LFC Portlet. The objectives of WRF4SG can be described by considering two goals. The first goal is to show how WRF4SG facilitates to execute, monitor and manage climate workflows based on the WRF4G framework. And the second goal of WRF4SG is to help WRF users to execute their experiment workflows concurrently using heterogeneous computing resources such as HPC and Grid.Peer reviewe

Benefits and requirements of grid computing for climate applications. An example with the community atmospheric model

13 páginas, 8 figuras.-- El Pdf del artículo es la versión pre-print.Grid computing is nowadays an established technology in fields such as High Energy Physics and Biomedicine, offering an alternative to traditional HPC for several problems; however, it is still an emerging discipline for the climate community and only a few climate applications have been adapted to the Grid to solve particular problems. In this paper we present an up-to-date description of the advantages and limitations of the Grid for climate applications (in particular global circulation models), analyzing the requirements and the new challenges posed to the Grid. In particular, we focus on production-like problems such as sensitivity analysis or ensemble prediction, where a single model is run several times with different parameters, forcing and/or initial conditions. As an illustrative example, we consider the Community Atmospheric Model (CAM) and analyze the advantages and shortcomings of the Grid to perform a sensitivity study of precipitation with SST perturbations in El Niño area, reporting the results obtained with traditional (local cluster) and Grid infrastructures. We conclude that new specific middleware (execution workflow managers) is needed to meet the particular requirements of climate applications (long simulations, checkpointing, etc.). This requires the side-by-side collaboration of IT and climate groups to deploy fully ported applications, such as the CAM for Grid (CAM4G) introduced in this paper.This work has benefited from the ESR VO infrastructure of the EU FP7 EGEE-III. This work has been partially supported by the EU FP7 project EELA-2 (Contract number 223797) and the Spanish Ministry of Science and Innovation through project CORWES (CGL2010-22158-C02-01).Peer reviewe

Execution management in the GRID, for sensitivity studies of global climate simulations

Open Access.Recent trends in climate modeling find in GRID computing a powerful way to achieve results by sharing geographically distributed computing and storage resources. In particular, ensemble prediction experiments are based on the generation of multiple model simulations to explore, statistically, the existing uncertainties in weather and climate forecast. In this paper, we present a GRID application consisting of a state-of-the-art climate model. The main goal of the application is to provide a tool that can be used by a climate researcher to run ensemble-based predictions on the GRID for sensitivity studies. One of the main duties of this tool is the management of a workflow involving long-term jobs and data management in a user-friendly way. In this paper we show that, due to weaknesses of current GRID middleware, this management is complex task. Those weaknesses made necessary the development of a robust workflow adapted to the requirements of the climate application. As an illustrative scientific challenge, the application is applied to study the El Niño phenomenon, by simulating an El Niño year with different forcing conditions and analyzing the precipitation response over south-American countries subject to flooding risk. © Springer-Verlag 2009.This work has been partially funded by the EELA project under the 6th Framework Program of the European Commission (contract no. 026409) and the Spanish Ministry of Education and Science through the Juan de la Cierva program.Peer Reviewe

Execution management in the GRID, for sensitivity studies of global climate simulations

Abstract Recent trends in climate modeling find in GRID computing a powerful way to achieve results by sharing geographically distributed computing and storage resources. In particular, ensemble prediction experiments are based on the generation of multiple model simulations to explore, statistically, the existing uncertainties in weather and climate forecast. In this paper, we present a GRID application consisting of a state-of-the-art climate model. The main goal of the application is to provide a tool that can be used by a climate researcher to run ensemble-based predictions on the GRID for sensitivity studies. One of the main duties of this tool is the management of a workflow involving long-term jobs and data management in a user-friendly way. In this paper we show that, due to weaknesses of current GRID middleware, this management is complex task. Those weaknesses made necessary the development of a robust workflow adapted to the requirements of the climate application. As an illustrative scientific challenge, the application is applied to study the El Niño phenomenon, by simulating an El Niño year with different forcing conditions and analyzing the precipitation response over south-American countries subject to flooding risk

Complex workflow management of the CAM global climate model on the GRID

Recent trends in climate modeling find in GRID computing a powerful way to achieve results by sharing computing and data distributed resources. In particular, ensemble prediction is based on the generation of multiple simulations from perturbed model conditions to sample the existing uncertainties. In this work, we present a GRID application consisting of a state-of-the-art climate model (CAM). The main goal of the application is providing a user-friendly platform to run ensemble-based predictions on the GRID. This requires managing a complex workflow involving long-term jobs and data management in a user-transparent way. In doing so, we identified the weaknesses of current GRID middleware tools and developed a robust workflow by merging the optimal existing applications with an underlying self-developed workflow.This work has been partial funded by the EELA project under the 6th Framework Program of the European Commission (contract no. 026409). J. F. is supported by the Spanish Ministry of Education and Science through the Juan de la Cierva program.Peer Reviewe

On the role of application and resource characterizations in heterogeneous distributed computing systems

Loosely coupled applications composed of a potentially very large number (from tens of thousands to even billions) of tasks are commonly used in high-throughput computing and many-task computing paradigms. To efficiently execute large-scale computations which can exceed the capability in a single type of computing resources within expected time, we should be able to effectively integrate resources from heterogeneous distributed computing (HDC) systems such as clusters, grids, and clouds. In this paper, we quantitatively analyze the performance of three different real scientific applications consisting of many tasks on top of HDC systems based on a partnership of distributed computing clusters, grids, and clouds to understand the application and resource characteristics, and show practical issues that normal scientific users can face during the course of leveraging these systems. Our experimental study shows that the performance of a loosely coupled application can be significantly affected by the characteristics of a HDC system, along with hardware specification of a node, and their impacts on the performance can vary widely depending on the resource usage pattern of each application. We then devise a preference-based scheduling algorithm that can reflect characteristics and resource usage patterns of various loosely coupled applications running on top of HDC systems from our experimental study. Our preference-based scheduling algorithm can allocate the resources from different HDC systems to loosely coupled applications based on the preferences of the applications for the HDC systems. We evaluate the overall system performance over various preference types, using trace-based simulations, which can be determined based on different factors such as CPU specifications and application throughputs. Our simulation results demonstrate the importance of understanding the application and resource characteristics on effective scheduling of loosely coupled applications on the HDC systems.clos