The PBase Scientific Workflow Provenance Repository

Scientific workflows and their supporting systems are becoming increasingly popular for compute-intensive and data-intensive scientific experiments. The advantages scientific workflows offer include rapid and easy workflow design, software and data reuse, scalable execution, sharing and collaboration, and other advantages that altogether facilitate “reproducible science”. In this context, provenance – information about the origin, context, derivation, ownership, or history of some artifact – plays a key role, since scientists are interested in examining and auditing the results of scientific experiments. However, in order to perform such analyses on scientific results as part of extended research collaborations, an adequate environment and tools are required. Concretely, the need arises for a repository that will facilitate the sharing of scientific workflows and their associated execution traces in an interoperable manner, also enabling querying and visualization. Furthermore, such functionality should be supported while taking performance and scalability into account. With this purpose in mind, we introduce PBase: a scientific workflow provenance repository implementing the ProvONE proposed standard, which extends the emerging W3C PROV standard for provenance data with workflow specific concepts. PBase is built on the Neo4j graph database, thus offering capabilities such as declarative and efficient querying. Our experiences demonstrate the power gained by supporting various types of queries for provenance data. In addition, PBase is equipped with a user friendly interface tailored for the visualization of scientific workflow provenance data, making the specification of queries and the interpretation of their results easier and more effective

WorkflowHunt : um mecanismo de busca híbrida para repositórios de workflows científicos

Orientador: Claudia Maria Bauzer MedeirosDissertação (mestrado) - Universidade Estadual de Campinas, Instituto de ComputaçãoResumo: Os experimentos científicos e os conjuntos de dados gerados a partir deles estão crescendo em tamanho e complexidade. Os cientistas estão enfrentando dificuldades para compartilhar esses recursos e permitir a reprodutibilidade do experimento. Algumas iniciativas surgiram para tentar resolver esse problema. Uma delas envolve o uso de workflows científicos para representar a execução de experimentos científicos. Existe um número crescente de workflows que são potencialmente relevantes para mais de um domínio científico. Criar um workflow leva tempo e recursos e sua reutilização ajuda aos cientistas a criar novos workflows de forma mais rápida e confiável. No entanto, é difícil encontrar workflows adequados para reutilização. Geralmente, os repositórios de workflows possuem mecanismos de busca com muitas limitações, o que afeta negativamente a descoberta de workflows relevantes para um cientista ou seu time. Esta dissertação apresenta WorkflowHunt, uma arquitetura híbrida para busca e descoberta de workflows em repositórios genéricos, combinando busca baseada em palavras-chave e busca semântica para encontrar workflows relevantes usando diferentes métodos de busca. Ao contrário da maioria das pesquisas correlatas, nossa proposta e sua implementação são genéricas. Nosso sistema de indexação e anotação é automático e independe de domínio ou ontologia específica. A arquitetura foi validada por meio de um protótipo que usa workflows e metadados reais do myExperiment, um dos maiores repositórios de workflows científicos. Nosso sistema também compara seus resultados com o mecanismo de busca do myExperiment para analisar em que casos um sistema supera o outroAbstract: Scientific experiments and the datasets generated from them are growing in size and complexity. Scientists are facing difficulties to share those resources in a way that allows reproducibility of the experiment. Some initiatives have emerged to try to solve this problem. One of them involves the use of scientific workflows to represent and enact the execution of scientific experiments. There is an increasing number of workflows that are potentially relevant for more than one scientific domain. Creating a workflow takes time and resources, and their reuse helps scientists to build new workflows faster and in a more reliable way. However, it is hard to find workflows suitable for reuse for an experiment. Usually, workflow repositories have search mechanisms with many limitations, which affects negatively the discovery of relevant workflows. This dissertation presents WorkflowHunt, a hybrid architecture for workflow search and discovery for generic repositories, which combines keyword and semantic search to find relevant workflows using different search methods. Unlike most related work, our proposal and its implementation are generic. Our indexing and annotation mechanism are automatic and not restricted to a specific domain or ontology. We validated our architecture creating a prototype that uses real workflows and metadata from myExperiment, one of the largest online scientific workflow repositories. Our system also compares its results with myExperiment¿s search engine to analyze in which cases one retrieval system outperforms the otherMestradoCiência da ComputaçãoMestre em Ciência da ComputaçãoCAPE

Workflow Provenance: from Modeling to Reporting

Workflow provenance is a crucial part of a workflow system as it enables data lineage analysis, error tracking, workflow monitoring, usage pattern discovery, and so on. Integrating provenance into a workflow system or modifying a workflow system to capture or analyze different provenance information is burdensome, requiring extensive development because provenance mechanisms rely heavily on the modelling, architecture, and design of the workflow system. Various tools and technologies exist for logging events in a software system. Unfortunately, logging tools and technologies are not designed for capturing and analyzing provenance information. Workflow provenance is not only about logging, but also about retrieving workflow related information from logs. In this work, we propose a taxonomy of provenance questions and guided by these questions, we created a workflow programming model 'ProvMod' with a supporting run-time library to provide automated provenance and log analysis for any workflow system. The design and provenance mechanism of ProvMod is based on recommendations from prominent research and is easy to integrate into any workflow system. ProvMod offers Neo4j graph database support to manage semi-structured heterogeneous JSON logs. The log structure is adaptable to any NoSQL technology. For each provenance question in our taxonomy, ProvMod provides the answer with data visualization using Neo4j and the ELK Stack. Besides analyzing performance from various angles, we demonstrate the ease of integration by integrating ProvMod with Apache Taverna and evaluate ProvMod usability by engaging users. Finally, we present two Software Engineering research cases (clone detection and architecture extraction) where our proposed model ProvMod and provenance questions taxonomy can be applied to discover meaningful insights

Research Data Curation and Management Bibliography

This e-book includes over 800 selected English-language articles and books that are useful in understanding the curation of digital research data in academic and other research institutions. It covers topics such as research data creation, acquisition, metadata, provenance, repositories, management, policies, support services, funding agency requirements, open access, peer review, publication, citation, sharing, reuse, and preservation. It has live links to included works. Abstracts are included in this bibliography if a work is under certain Creative Commons Licenses. This book is licensed under a Creative Commons Attribution 4.0 International License. Cite as: Bailey, Charles W., Jr. Research Data Curation and Management Bibliography. Houston: Digital Scholarship, 2021