1 research outputs found
Evaluation of pilot jobs for Apache Spark applications on HPC clusters
Big Data has become prominent throughout many scientific fields and, as a
result, scientific communities have sought out Big Data frameworks to
accelerate the processing of their increasingly data-intensive pipelines.
However, while scientific communities typically rely on High-Performance
Computing (HPC) clusters for the parallelization of their pipelines, many
popular Big Data frameworks such as Hadoop and Apache Spark were primarily
designed to be executed on dedicated commodity infrastructures. This paper
evaluates the benefits of pilot jobs over traditional batch submission for
Apache Spark on HPC clusters. Surprisingly, our results show that the speed-up
provided by pilot jobs over batch scheduling is moderate to inexistent (0.98 on
average) despite the presence of long queuing times. In addition, pilot jobs
provide an extra layer of scheduling that complexifies debugging and
deployment. We conclude that traditional batch scheduling should remain the
default strategy to deploy Apache Spark applications on HPC clusters