4 research outputs found
A performance comparison of Dask and Apache Spark for data-intensive neuroimaging pipelines
In the past few years, neuroimaging has entered the Big Data era due to the
joint increase in image resolution, data sharing, and study sizes. However, no
particular Big Data engines have emerged in this field, and several
alternatives remain available. We compare two popular Big Data engines with
Python APIs, Apache Spark and Dask, for their runtime performance in processing
neuroimaging pipelines. Our evaluation uses two synthetic pipelines processing
the 81GB BigBrain image, and a real pipeline processing anatomical data from
more than 1,000 subjects. We benchmark these pipelines using various
combinations of task durations, data sizes, and numbers of workers, deployed on
an 8-node (8 cores ea.) compute cluster in Compute Canada's Arbutus cloud. We
evaluate PySpark's RDD API against Dask's Bag, Delayed and Futures. Results
show that despite slight differences between Spark and Dask, both engines
perform comparably. However, Dask pipelines risk being limited by Python's GIL
depending on task type and cluster configuration. In all cases, the major
limiting factor was data transfer. While either engine is suitable for
neuroimaging pipelines, more effort needs to be placed in reducing data
transfer time.Comment: 10 pages, 15 figures, 1 tables. To appear in the proceeding of the
14th WORKS Workshop on Topics in Workflows in Support of Large-Scale Science,
17 November 2019, Denver, CO, US
Big data workflows: Locality-aware orchestration using software containers
The emergence of the Edge computing paradigm has shifted data processing from centralised infrastructures to heterogeneous and geographically distributed infrastructures. Therefore, data processing solutions must consider data locality to reduce the performance penalties from data transfers among remote data centres. Existing Big Data processing solutions provide limited support for handling data locality and are inefficient in processing small and frequent events specific to the Edge environments. This article proposes a novel architecture and a proof-of-concept implementation for software container-centric Big Data workflow orchestration that puts data locality at the forefront. The proposed solution considers the available data locality information, leverages long-lived containers to execute workflow steps, and handles the interaction with different data sources through containers. We compare the proposed solution with Argo Workflows and demonstrate a significant performance improvement in the execution speed for processing the same data units. Finally, we carry out experiments with the proposed solution under different configurations and analyze individual aspects affecting the performance of the overall solution.publishedVersio
Big data workflows: Locality-aware orchestration using software containers
The emergence of the Edge computing paradigm has shifted data processing from centralised infrastructures to heterogeneous and geographically distributed infrastructures. Therefore, data processing solutions must consider data locality to reduce the performance penalties from data transfers among remote data centres. Existing Big Data processing solutions provide limited support for handling data locality and are inefficient in processing small and frequent events specific to the Edge environments. This article proposes a novel architecture and a proof-of-concept implementation for software container-centric Big Data workflow orchestration that puts data locality at the forefront. The proposed solution considers the available data locality information, leverages long-lived containers to execute workflow steps, and handles the interaction with different data sources through containers. We compare the proposed solution with Argo Workflows and demonstrate a significant performance improvement in the execution speed for processing the same data units. Finally, we carry out experiments with the proposed solution under different configurations and analyze individual aspects affecting the performance of the overall solution.publishedVersio