1 research outputs found
Exploring Trade-offs in Dynamic Task Triggering for Loosely Coupled Scientific Workflows
In order to achieve near-time insights, scientific workflows tend to be
organized in a flexible and dynamic way. Data-driven triggering of tasks has
been explored as a way to support workflows that evolve based on the data.
However, the overhead introduced by such dynamic triggering of tasks is an
under-studied topic. This paper discusses different facets of dynamic task
triggers. Particularly, we explore different ways of constructing a data-driven
dynamic workflow and then evaluate the overheads introduced by such design
decisions. We evaluate workflows with varying data size, percentage of
interesting data, temporal data distribution, and number of tasks triggered.
Finally, we provide advice based upon analysis of the evaluation results for
users looking to construct data-driven scientific workflows