1 research outputs found
Smart Resource Management for Data Streaming using an Online Bin-packing Strategy
Data stream processing frameworks provide reliable and efficient mechanisms
for executing complex workflows over large datasets. A common challenge for the
majority of currently available streaming frameworks is efficient utilization
of resources. Most frameworks use static or semi-static settings for resource
utilization that work well for established use cases but lead to marginal
improvements for unseen scenarios. Another pressing issue is the efficient
processing of large individual objects such as images and matrices typical for
scientific datasets. HarmonicIO has proven to be a good solution for streams of
relatively large individual objects, as demonstrated in a benchmark comparison
with the Spark and Kafka streaming frameworks. We here present an extension of
the HarmonicIO framework based on the online bin-packing algorithm, to allow
for efficient utilization of resources. Based on a real world use case from
large-scale microscopy pipelines, we compare results of the new system to
Spark's auto-scaling mechanism