Accelerating complex modeling workflows in CyberWater using on-demand HPC/Cloud resources

Workflow management systems (WMSs) are commonly used to organize/automate sequences of tasks as workflows to accelerate scientific discoveries. During complex workflow modeling, a local interactive workflow environment is desirable, as users usually rely on their rich, local environments for fast prototyping and refinements before they consider using more powerful computing resources. However, existing WMSs do not simultaneously support local interactive workflow environments and HPC resources. In this paper, we present an on-demand access mechanism to remote HPC resources from desktop/laptop-based workflow management software to compose, monitor and analyze scientific workflows in the CyberWater project. Cyber-Water is an open-data and open-modeling software framework for environmental and water communities. In this work, we extend the open-model, open-data design of CyberWater with on-demand HPC accessing capacity. In particular, we design and implement the LaunchAgent library, which can be integrated into the local desktop environment to allow on-demand usage of remote resources for hydrology-related workflows. LaunchAgent manages authentication to remote resources, prepares the computationally-intensive or data-intensive tasks as batch jobs, submits jobs to remote resources, and monitors the quality of services for the users. LaunchAgent interacts seamlessly with other existing components in CyberWater, which is now able to provide advantages of both feature-rich desktop software experience and increased computation power through on-demand HPC/Cloud usage. In our evaluations, we demonstrate how a hydrology workflow that consists of both local and remote tasks can be constructed and show that the added on-demand HPC/Cloud usage helps speeding up hydrology workflows while allowing intuitive workflow configurations and execution using a desktop graphical user interface

Cluster Management for non-XSEDE Systems

A webinar hosted by XSEDE and IU discussing no-cost, open source resources for scientific computation.XSED

GridChem and ParamChem: Science Gateways for Computational Chemistry (and More)

Presented at HUBBUB 2012, the HUBZero conference, 24-25 September 2012. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the NSF.Support for GridChem/ParamChem integration with Apache Airavata from NSF OCI 1032742 - SDCI NMI Improvement: Open Gateway Computing Environment

Better Data Discoverability in Science Gateways

Science gateways primarily focused on remote job executionmanagement generate domain specific output data mainlyreadable by application specific parsers and post processing utilities. For example, computational chemistry data outputs encode molecule information, convergence of the simulation and energy values. Such domain-specific information is non-trivial to search in a generic fashion. It is thus desirable to add a wide range of application-specific and user-specific post-processing features that may include remote executions of scripts and smaller applications that don’t require scheduling on clusters. It is also desirable to support integrations with searching, indexing, and general purpose data analysis and mining tools provided by the Apache “big data” software stack. As gateways become tenants to general purpose platform services, providing a general purpose infrastructure that enables these application specific post-processing steps is an interesting architectural challenge. Furthermore, it is desirable to share results fromthe post-processing and indexing. In this paper, we discuss how we have incorporated a new automated application output indexing system for the SEAGrid Science Gateway using Apache Airavata that will parse and index generated output for easy querying. We also examine data sharing and automated data publication so that another user can reuse theresults without running an already executed experiment andhence reduce resource utilization

From Proposal to Production: Lessons Learned Developing the Computational Chemistry Grid Cyberinfrastructure

Abstract. The Computational Chemistry Grid (CCG) is a 3-year, National Middleware Initiative (NMI) program to develop cyberinfrastructure for the chemistry community. CCG is led by the University of Kentucky, and involves collaboratin