Design considerations for workflow management systems use in production genomics research and the clinic

Abstract The changing landscape of genomics research and clinical practice has created a need for computational pipelines capable of efficiently orchestrating complex analysis stages while handling large volumes of data across heterogeneous computational environments. Workflow Management Systems (WfMSs) are the software components employed to fill this gap. This work provides an approach and systematic evaluation of key features of popular bioinformatics WfMSs in use today: Nextflow, CWL, and WDL and some of their executors, along with Swift/T, a workflow manager commonly used in high-scale physics applications. We employed two use cases: a variant-calling genomic pipeline and a scalability-testing framework, where both were run locally, on an HPC cluster, and in the cloud. This allowed for evaluation of those four WfMSs in terms of language expressiveness, modularity, scalability, robustness, reproducibility, interoperability, ease of development, along with adoption and usage in research labs and healthcare settings. This article is trying to answer, which WfMS should be chosen for a given bioinformatics application regardless of analysis type?. The choice of a given WfMS is a function of both its intrinsic language and engine features. Within bioinformatics, where analysts are a mix of dry and wet lab scientists, the choice is also governed by collaborations and adoption within large consortia and technical support provided by the WfMS team/community. As the community and its needs continue to evolve along with computational infrastructure, WfMSs will also evolve, especially those with permissive licenses that allow commercial use. In much the same way as the dataflow paradigm and containerization are now well understood to be very useful in bioinformatics applications, we will continue to see innovations of tools and utilities for other purposes, like big data technologies, interoperability, and provenance

Modelagem e simulação de sistemas de gerenciamento de processos de negócios baseadas em workflow net temporais com mecanismos de alocação de recursos híbridos fuzzy

The main objective of this doctoral research is to propose a new business process model for efficient (and realistic) resources and time management in Workflow nets. This research had as its starting point the validation of the fuzzy resource allocation model presented in (JESKE, 2006). In particular, a specialized inference mechanism of the token player type adapted to a fuzzy hybrid resource allocation model was implemented in the modeling, analysis and simulation tool of colored Petri nets called CPN Tools. Such simulations have shown that the new model in relation to a simple resource allocation mechanism with an earlier firing policy (FIFO) allows an improvement in the proportion of cases that meet the established deadlines. To solve the scheduling problem in Workflow Management Systems, important characteristics inherent to these systems should be considered. In particular, when many cases are run concurrently, conflict situations due to shared resource can occur and must be resolved in real time (without a backward mechanism). Also, temporary restrictions regarding the deadlines for delivery of specific cases should be considered. For that, a fuzzy time associated to the activities of the processes was defined as well as a mechanism of propagation of restrictions whose role is to guarantee the consistency of the time constraints to carry out the activities respecting the expected delivery times. With the objective of making the behavior of the new model the closest and consistent with the behavior of the human collaborators involved in the execution of business process activities, the notion of joint possibility was inserted in the model. This notion involves the uncertain and nebulous character of the execution times of the activities of the processes and the intensity of the resources used to carry out the activities. New firing rules for transitions were then defined based on the concept presented. Token players based on the new model presented were defined, implemented, simulated and validated in the CPN Tools, whose main role is to show the execution of the new models presented. Using the CPN Tools, it was possible to evaluate the system behavior considering the expected or most critical operating. It is important to point out that through the simulation results it is possible to predict how the system will behave and, from this, to evaluate several possible resource distribution scenarios in order to be able, in the future, to carry out the sizing of the system based on a modeling and simulation tool that approximates of the way employees work in an office environment.Tese (Doutorado)O principal objetivo desta pesquisa de doutorado é propor um novo modelo de processo de negócios para uma gestão eficiente (e realista) dos recursos e do tempo em Workflow nets. Esta pesquisa teve como ponto de partida a validação do modelo de alocação de recursos fuzzy apresentado em (JESKE, 2006). Em particular, um mecanismo de inferência especializado do tipo token player adaptado a um modelo de alocação de recurso híbrido fuzzy foi implementado na ferramenta de modelagem, análise e simulação de redes de Petri coloridas chamada CPN Tools. Tais simulações mostraram que o novo modelo em relação a um mecanismo simples de alocação de recurso com uma política de disparo mais cedo (FIFO) permite uma melhoria da proporção de casos que respeitam os prazos estabelecidos. Para resolver o problema do escalonamento em Sistemas de Gerenciamento de Workflow deverão ser consideradas características importantes inerentes a estes sistemas. Em particular, quando muitos casos são executados simultaneamente, situações de conflito por recursos compartilhados podem ocorrer e devem ser resolvidas em tempo real (sem um mecanismo de retrocesso). Também, deverão ser consideradas restrições temporais relativas aos prazos de entrega de casos específicos. Para tanto, definiu-se um tempo fuzzy associado às atividades dos processos bem como um mecanismo de propagação de restrições cujo papel é garantir a consistência das restrições de tempo para realização das atividades respeitando os prazos de entrega previstos. Com o objetivo de tornar o comportamento do novo modelo o mais próximo e consistente com o comportamento de funcionários humanos envolvidos na execução de atividades de processos de negócios, a noção de possibilidade conjunta que envolve o caráter incerto e nebuloso tanto dos tempos de execução das atividades dos processos quanto das intensidades de carga dos recursos utilizados para a realização de atividades foi inserida no modelo. Novas regras de disparo de transições foram então definidas baseadas no conceito apresentado. Jogadores de rede de Petri do tipo token player baseados no novo modelo apresentado foram definidos, implementados, simulados e validados na ferramenta CPN Tools, cujo papel principal é mostrar a execução dos novos modelos apresentados. Com o uso da ferramenta CPN Tools foi possível avaliar o comportamento do sistema em funcionamento esperado ou em situação mais crítica. É importante salientar que através dos resultados de simulação é possível prever como o sistema se comportará, isso a fim de avaliar diversos cenários possíveis de distribuição de recursos para poder, no futuro, realizar o dimensionamento do sistema baseado numa ferramenta de modelagem e simulação próxima da forma de trabalhar dos funcionários em ambiente de escritório