Heterogeneous hierarchical workflow composition

Workflow systems promise scientists an automated end-to-end path from hypothesis to discovery. However, expecting any single workflow system to deliver such a wide range of capabilities is impractical. A more practical solution is to compose the end-to-end workflow from more than one system. With this goal in mind, the integration of task-based and in situ workflows is explored, where the result is a hierarchical heterogeneous workflow composed of subworkflows, with different levels of the hierarchy using different programming, execution, and data models. Materials science use cases demonstrate the advantages of such heterogeneous hierarchical workflow composition.This work is a collaboration between Argonne National Laboratory and the Barcelona Supercomputing Center within the Joint Laboratory for Extreme-Scale Computing. This research is supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research, under contract number DE-AC02- 06CH11357, program manager Laura Biven, and by the Spanish Government (SEV2015-0493), by the Spanish Ministry of Science and Innovation (contract TIN2015-65316-P), by Generalitat de Catalunya (contract 2014-SGR-1051).Peer ReviewedPostprint (author's final draft

Chiminey: Reliable Computing and Data Management Platform in the Cloud

The enabling of scientific experiments that are embarrassingly parallel, long running and data-intensive into a cloud-based execution environment is a desirable, though complex undertaking for many researchers. The management of such virtual environments is cumbersome and not necessarily within the core skill set for scientists and engineers. We present here Chiminey, a software platform that enables researchers to (i) run applications on both traditional high-performance computing and cloud-based computing infrastructures, (ii) handle failure during execution, (iii) curate and visualise execution outputs, (iv) share such data with collaborators or the public, and (v) search for publicly available data.Comment: Preprint, ICSE 201

Software Challenges For HL-LHC Data Analysis

The high energy physics community is discussing where investment is needed to prepare software for the HL-LHC and its unprecedented challenges. The ROOT project is one of the central software players in high energy physics since decades. From its experience and expectations, the ROOT team has distilled a comprehensive set of areas that should see research and development in the context of data analysis software, for making best use of HL-LHC's physics potential. This work shows what these areas could be, why the ROOT team believes investing in them is needed, which gains are expected, and where related work is ongoing. It can serve as an indication for future research proposals and cooperations

SimStack: An Intuitive Workflow Framework

Establishing a fundamental understanding of the nature of materials via computational simulation approaches requires knowledge from different areas, including physics, materials science, chemistry, mechanical engineering, mathematics, and computer science. Accurate modeling of the characteristics of a particular system usually involves multiple scales and therefore requires the combination of methods from various fields into custom-tailored simulation workflows. The typical approach to developing patch-work solutions on a case-to-case basis requires extensive expertise in scripting, command-line execution, and knowledge of all methods and tools involved for data preparation, data transfer between modules, module execution, and analysis. Therefore multiscale simulations involving state-of-the-art methods suffer from limited scalability, reproducibility, and flexibility. In this work, we present the workflow framework SimStack that enables rapid prototyping of simulation workflows involving modules from various sources. In this platform, multiscale- and multimodule workflows for execution on remote computational resources are crafted via drag and drop, minimizing the required expertise and effort for workflow setup. By hiding the complexity of high-performance computations on remote resources and maximizing reproducibility, SimStack enables users from academia and industry to combine cutting-edge models into custom-tailored, scalable simulation solutions

PyCOMPSs as an instrument for translational computer science

With the advent of distributed computing, the need for frameworks that facilitate its programming and management has also appeared. These tools have typically been used to support the research on application areas that require them. This poses good initial conditions for translational computer science (TCS), although this does not always occur. This article describes our experience with the PyCOMPSs project, a programming model for distributed computing. While it is a research instrument for our team, it has also been applied in multiple real use cases under the umbrella of European Funded projects, or as part of internal projects between various departments at the Barcelona Supercomputing Center. This article illustrates how the authors have engaged in TCS as an underlying research methodology, collecting experiences from three European projects.This work was supported in part by Spanish Government under Contract TIN2015-65316-P, in part by the Generalitat de Catalunya under Contract 2014-SGR-1051, and in part by the European Commission’s Horizon 2020 Framework program through BioExcel Center of Excellence under Contract 823830 and Contract 675728, in part by the ExaQUte Project under Contract 800898, in part by the European High-Performance Computing Joint Undertaking (JU) under Grant 955558, in part by the MCIN/AEI/10.13039/501100011033, and in part by the European Union NextGenerationEU/PRTR.Peer ReviewedPostprint (author's final draft