Heterogeneous hierarchical workflow composition

Workflow systems promise scientists an automated end-to-end path from hypothesis to discovery. However, expecting any single workflow system to deliver such a wide range of capabilities is impractical. A more practical solution is to compose the end-to-end workflow from more than one system. With this goal in mind, the integration of task-based and in situ workflows is explored, where the result is a hierarchical heterogeneous workflow composed of subworkflows, with different levels of the hierarchy using different programming, execution, and data models. Materials science use cases demonstrate the advantages of such heterogeneous hierarchical workflow composition.This work is a collaboration between Argonne National Laboratory and the Barcelona Supercomputing Center within the Joint Laboratory for Extreme-Scale Computing. This research is supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research, under contract number DE-AC02- 06CH11357, program manager Laura Biven, and by the Spanish Government (SEV2015-0493), by the Spanish Ministry of Science and Innovation (contract TIN2015-65316-P), by Generalitat de Catalunya (contract 2014-SGR-1051).Peer ReviewedPostprint (author's final draft

HPC Cloud for Scientific and Business Applications: Taxonomy, Vision, and Research Challenges

High Performance Computing (HPC) clouds are becoming an alternative to on-premise clusters for executing scientific applications and business analytics services. Most research efforts in HPC cloud aim to understand the cost-benefit of moving resource-intensive applications from on-premise environments to public cloud platforms. Industry trends show hybrid environments are the natural path to get the best of the on-premise and cloud resources---steady (and sensitive) workloads can run on on-premise resources and peak demand can leverage remote resources in a pay-as-you-go manner. Nevertheless, there are plenty of questions to be answered in HPC cloud, which range from how to extract the best performance of an unknown underlying platform to what services are essential to make its usage easier. Moreover, the discussion on the right pricing and contractual models to fit small and large users is relevant for the sustainability of HPC clouds. This paper brings a survey and taxonomy of efforts in HPC cloud and a vision on what we believe is ahead of us, including a set of research challenges that, once tackled, can help advance businesses and scientific discoveries. This becomes particularly relevant due to the fast increasing wave of new HPC applications coming from big data and artificial intelligence.Comment: 29 pages, 5 figures, Published in ACM Computing Surveys (CSUR

FLOWViZ: framework for phylogenetic processing

Final project to obtain the Master Degree in Computer Science and EngineeringO aumento do risco epidemiológico e o constante crescimento da população mundial contribuiu para que se fizesse um forte investimento na análise filogenética, de modo a monitorizar doenças e a conceber tratamentos e medicação rápidos e eficazes. A análise filogenética utiliza grandes quantidades de informação, que deve ser analisada e processada para se extrair conhecimento, utilizando técnicas adequadas e, atualmente, software especializado e algoritmos, de modo a produzir resultados eficazes e rápidos. Estes algoritmos já são fornecidos por um grande conjunto de frameworks e ferramentas disponíveis gratuitamente, um bom exemplo é a framework de inferência filogenética PHYLOViZ[23]. A maioria das técnicas de análise utilizadas na inferência filogenética tendem a formar, topologicamente, pipelines de trabalho - procedimentos constituídos por passos, cujos fluxos de dados são dependentes entre si. Apesar de ser possível executar pipelines de trabalho manualmente, como tem sido feito há várias décadas, atualmente, já não é fazível, dado que os datasets utilizados são volumosos, tornando a sua análise manual contraproducente. A transição manual entre passos necessita também que haja interação humana para que cada passo receba os dados necessários, o que pode também estar sujeito ao erro humano. Por isso, foi construído software que reduzisse a interação humana e que automatizasse estes procedimentos. Este tipo de software é designado por sistemas de workflow - software que permite os utilizadores criarem workflows, através de uma Domain-Specific Language (DSL)[13], onde estes procedimentos são traduzidos para scripts, especificandose o grupo de tarefas, com os seus parâmetros e dependências de dados. Existem atualmente várias soluções de sistemas de workflow, que diferem na sua linguagem e estruturação de workflows, o que leva a que exista uma grande heterogeneidade de software, mas que piora também a partilha destes procedimentos. Por isso, quando se partilham workflows, é necessário despender-se tempo a traduzir pipelines de trabalho para a linguagem específica do sistema de workflow que vai executar a pipeline partilhada. Este problema levou a que fosse criada a Common Workflow Language (CWL)[2] - um novo standard que permite executar workflows entre vários sistemas de workflow. No entanto, nem todos os sistemas suportam este novo standard. Este projeto pretende construir uma framework, recorrendo a um projeto existente - PHYLOViZ e ao seu conjunto de ferramentas de inferência filogenética. Esta framework, permitirá ligar frameworks de inferência filogenética a sistemas de workflow, dando ao utilizador liberdade para construir os seus workflows personalizados, recorrendo à framework e às ferramentas do utilizador, fornecidas remotamente, que poderão ser geridas através de uma interface intuitiva. Tudo isto, fornecerá automatização de workflows e uma análise filogenética mais rápida e eficaz. Este projeto foi financiado, no contexto de uma bolsa de estudo da Fundação para a Ciência e a Tecnologia (FCT) com referência UIDB/50021/2020, no projeto NGPHYLO PTDC/CCI-BIO/29676/2017 e num projeto do IPL - IPL/2021/DIVA_ISEL.The increasing risk of epidemics and a fast-growing world population has contributed to a great investment in phylogenetic analysis, in order to track numerous diseases and conceive effective medication and treatments. Phylogenetic analysis requires large quantities of information to be analyzed and processed for knowledge extraction, using adequate techniques and, nowadays, specific software and algorithms, to deliver results as efficiently and fast as possible. These algorithms and techniques are already provided by a great set of free and available frameworks and tools, such as PHYLOViZ[23]. Most of the applied techniques and algorithms used for phylogenetic inference tend to form work pipelines - procedures formed by steps, which typically have an intrinsic dependency between them. Although it is possible to execute work pipelines manually, as it has been done for decades, nowadays, is not feasible, as genomic datasets are very large, and the respective analysis is time-consuming. The transition between steps also needs human interaction and each step must receive the matching data, correctly, which can introduce human error. Because of this, software were made to ease and reduce manual interaction, so these procedures could be automated. This type of software is typically referred as a workflow system - software which allows users to create workflows, on top of a provided Domain-Specific Language (DSL)[13], where procedures are translated into scripts, through the definition of a group of steps and their specific parameters and dependencies. There are already many software solutions available, which differ in their Domain-Specific Language and workflow structuring, leading to a great software heterogeneity, but also low workflow shareability - as users work on different workflow systems. Thus, when they share workflows with others, time needs to be spent converting and adapting certain workflows to a specific workflow system, so work pipelines can be executed, making workflow sharing a difficult task. This lead to the creation of the Common Workflow Language (CWL)[2] - a new standard which provides a way to execute workflows and work pipelines among diferente workflow systems. However, not every system supports this new standard. This project aims to build a framework on top of an already existing project - PHYLOViZ, which provides a set of state-of-the-art tools for phylogenetic inference. The developed framework, will link phylogenetic inference web frameworks with workflow systems, giving the user freedom to build its workflows, using the provided web framework’s or its remote tools, through a user-friendly web interface. Resulting in workflow automation, task scheduling and a more efficient and faster phylogenetic analysis. The project was supported by funds, under the context of a student grant of Fundação para a Ciência e a Tecnologia (FCT) with reference UIDB/50021/2020, for a INESCID’s project - NGPHYLO PTDC/CCI-BIO/29676/2017 and a Polytechnic Institute of Lisbon project - IPL/2021/DIVA_ISEL.N/

OperA/ALIVE/OperettA

Comprehensive models for organizations must, on the one hand, be able to specify global goals and requirements but, on the other hand, cannot assume that particular actors will always act according to the needs and expectations of the system design. Concepts as organizational rules (Zambonelli 2002), norms and institutions (Dignum and Dignum 2001; Esteva et al. 2002), and social structures (Parunak and Odell 2002) arise from the idea that the effective engineering of organizations needs high-level, actor-independent concepts and abstractions that explicitly define the organization in which agents live (Zambonelli 2002).Peer ReviewedPostprint (author's final draft

Automatically generating complex test cases from simple ones

While source code expresses and implements design considerations for software system, test cases capture and represent the domain knowledge of software developer, her assumptions on the implicit and explicit interaction protocols in the system, and the expected behavior of different modules of the system in normal and exceptional conditions. Moreover, test cases capture information about the environment and the data the system operates on. As such, together with the system source code, test cases integrate important system and domain knowledge. Besides being an important project artifact, test cases embody up to the half the overall software development cost and effort. Software projects produce many test cases of different kind and granularity to thoroughly check the system functionality, aiming to prevent, detect, and remove different types of faults. Simple test cases exercise small parts of the system aiming to detect faults in single modules. More complex integration and system test cases exercise larger parts of the system aiming to detect problems in module interactions and verify the functionality of the system as a whole. Not surprisingly, the test case complexity comes at a cost -- developing complex test cases is a laborious and expensive task that is hard to automate. Our intuition is that important information that is naturally present in test cases can be reused to reduce the effort in generation of new test cases. This thesis develops this intuition and investigates the phenomenon of information reuse among test cases. We first empirically investigated many test cases from real software projects and demonstrated that test cases of different granularity indeed share code fragments and build upon each other. Then we proposed an approach for automatically generating complex test cases by extracting and exploiting information in existing simple ones. In particular, our approach automatically generates integration test cases from unit ones. We implemented our approach in a prototype to evaluate its ability to generate new and useful test cases for real software systems. Our studies show that test cases generated with our approach reveal new interaction faults even in well tested applications. We evaluated the effectiveness of our approach by comparing it with the state of the art test generation techniques. The evaluation results show that our approach is effective, it finds relevant faults differently from other approaches that tend to find different and usually less relevant faults

Interaction Expressions - A Powerful Formalism for Describing Inter-Workflow Dependencies

Current workflow management technology does not provide adequate means for describing and implementing workflow ensembles, i. e., dynamically evolving collections of more or less independent workflows which have to synchronize only now and then. Interaction expressions are proposed as a simple yet powerful formalism to remedy this shortcoming. Besides operators for sequential composition, iteration, and selection, which are well-known from regular expressions, they provide parallel composition and iteration, conjunction, and several advanced features like parametric expressions, multipliers, and quantifiers. The paper introduces interaction expressions semi-formally, gives examples of their typical use, and describes their implementation and integration with state-of-the-art workflow technology. Major design principles, such as orthogonality and implicit and predictive choice, are discussed and compared with several related approaches

Full contract verification for ATL using symbolic execution

The Atlas Transformation Language (ATL) is currently one of the most used model transformation languages and has become a de facto standard in model-driven engineering for implementing model transformations. At the same time, it is understood by the community that enhancing methods for exhaustively verifying such transformations allows for a more widespread adoption of model-driven engineering in industry. A variety of proposals for the verification of ATL transformations have arisen in the past few years. However, the majority of these techniques are either based on non-exhaustive testing or on proof methods that require human assistance and/or are not complete. In this paper, we describe our method for statically verifying the declarative subset of ATL model transformations. This verification is performed by translating the transformation (including features like filters, OCL expressions, and lazy rules) into our model transformation language DSLTrans. As we handle only the declarative portion of ATL, and DSLTrans is Turing-incomplete, this reduction in expressivity allows us to use a symbolic-execution approach to generate representations of all possible input models to the transformation. We then verify pre-/post-condition contracts on these representations, which in turn verifies the transformation itself. The technique we present in this paper is exhaustive for the subset of declarative ATL model transformations. This means that if the prover indicates a contract holds on a transformation, then the contract’s pre-/post-condition pair will be true for any input model for that transformation. We demonstrate and explore the applicability of our technique by studying several relatively large and complex ATL model transformations, including a model transformation developed in collaboration with our industrial partner. As well, we present our ‘slicing’ technique. This technique selects only those rules in the DSLTrans transformation needed for contract proof, thereby reducing proving timeComisión Interministerial de Ciencia y Tecnología TIN2015-70560-RJunta de Andalucía P10-TIC-5906Junta de Andalucía P12-TIC-186

Bridging the Gap Between Requirements and Model Analysis : Evaluation on Ten Cyber-Physical Challenge Problems

Formal verfication and simulation are powerful tools to validate requirements against complex systems. [Problem] Requirements are developed in early stages of the software lifecycle and are typically written in ambiguous natural language. There is a gap between such requirements and formal notations that can be used by verification tools, and lack of support for proper association of requirements with software artifacts for verification. [Principal idea] We propose to write requirements in an intuitive, structured natural language with formal semantics, and to support formalization and model/code verification as a smooth, well-integrated process. [Contribution] We have developed an end-to-end, open source requirements analysis framework that checks Simulink models against requirements written in structured natural language. Our framework is built in the Formal Requirements Elicitation Tool (fret); we use fret's requirements language named fretish, and formalization of fretish requirements in temporal logics. Our proposed framework contributes the following features: 1) automatic extraction of Simulink model information and association of fretish requirements with target model signals and components; 2) translation of temporal logic formulas into synchronous dataflow cocospec specifications as well as Simulink monitors, to be used by verification tools; we establish correctness of our translation through extensive automated testing; 3) interpretation of counterexamples produced by verification tools back at requirements level. These features support a tight integration and feedback loop between high level requirements and their analysis. We demonstrate our approach on a major case study: the Ten Lockheed Martin Cyber-Physical, aerospace-inspired challenge problems