4 research outputs found
BigDL: A Distributed Deep Learning Framework for Big Data
This paper presents BigDL (a distributed deep learning framework for Apache
Spark), which has been used by a variety of users in the industry for building
deep learning applications on production big data platforms. It allows deep
learning applications to run on the Apache Hadoop/Spark cluster so as to
directly process the production data, and as a part of the end-to-end data
analysis pipeline for deployment and management. Unlike existing deep learning
frameworks, BigDL implements distributed, data parallel training directly on
top of the functional compute model (with copy-on-write and coarse-grained
operations) of Spark. We also share real-world experience and "war stories" of
users that have adopted BigDL to address their challenges(i.e., how to easily
build end-to-end data analysis and deep learning pipelines for their production
data).Comment: In ACM Symposium of Cloud Computing conference (SoCC) 201
The Challenges of In Situ Analysis for Multiple Simulations
International audienceIn situ analysis and visualization have mainly been applied to the output of a single large-scale simulation. However, topics involving the execution of multiple simulations in supercomputers have only received minimal attention so far. Some important examples are uncertainty quantification, data assimilation, and complex optimization. In this position article, beyond highlighting the strengths and limitations of the tools that we have developed over the past few years, we share lessons learned from using them on large-scale platforms and from interacting with end users. We then discuss the forthcoming challenges, which future in situ analysis and vi-sualization frameworks will face when dealing with the exascale execution of multiple simulations