Scientific workflow management systems (SWMSs) and resource managers together
ensure that tasks are scheduled on provisioned resources so that all
dependencies are obeyed, and some optimization goal, such as makespan
minimization, is fulfilled. In practice, however, there is no clear separation
of scheduling responsibilities between an SWMS and a resource manager because
there exists no agreed-upon separation of concerns between their different
components. This has two consequences. First, the lack of a standardized API to
exchange scheduling information between SWMSs and resource managers hinders
portability. It incurs costly adaptations when a component should be replaced
by another one (e.g., an SWMS with another SWMS on the same resource manager).
Second, due to overlapping functionalities, current installations often
actually have two schedulers, both making partial scheduling decisions under
incomplete information, leading to suboptimal workflow scheduling.
In this paper, we propose a simple REST interface between SWMSs and resource
managers, which allows any SWMS to pass dynamic workflow information to a
resource manager, enabling maximally informed scheduling decisions. We provide
an exemplary implementation of this API for Nextflow as an SWMS and Kubernetes
as a resource manager. Our experiments with nine real-world workflows show that
this strategy reduces makespan by up to 25.1% and 10.8% on average compared to
the standard Nextflow/Kubernetes configuration. Furthermore, a more widespread
implementation of this API would enable leaner code bases, a simpler exchange
of components of workflow systems, and a unified place to implement new
scheduling algorithms.Comment: Paper accepted in: 2023 23rd IEEE International Symposium on Cluster,
Cloud and Internet Computing (CCGrid