With the increasing amount of data available to scientists in disciplines as
diverse as bioinformatics, physics, and remote sensing, scientific workflow
systems are becoming increasingly important for composing and executing
scalable data analysis pipelines. When writing such workflows, users need to
specify the resources to be reserved for tasks so that sufficient resources are
allocated on the target cluster infrastructure. Crucially, underestimating a
task's memory requirements can result in task failures. Therefore, users often
resort to overprovisioning, resulting in significant resource wastage and
decreased throughput.
In this paper, we propose a novel online method that uses monitoring time
series data to predict task memory usage in order to reduce the memory wastage
of scientific workflow tasks. Our method predicts a task's runtime, divides it
into k equally-sized segments, and learns the peak memory value for each
segment depending on the total file input size. We evaluate the prototype
implementation of our method using workflows from the publicly available
nf-core repository, showing an average memory wastage reduction of 29.48%
compared to the best state-of-the-art approac