6,176 research outputs found
What does fault tolerant Deep Learning need from MPI?
Deep Learning (DL) algorithms have become the de facto Machine Learning (ML)
algorithm for large scale data analysis. DL algorithms are computationally
expensive - even distributed DL implementations which use MPI require days of
training (model learning) time on commonly studied datasets. Long running DL
applications become susceptible to faults - requiring development of a fault
tolerant system infrastructure, in addition to fault tolerant DL algorithms.
This raises an important question: What is needed from MPI for de- signing
fault tolerant DL implementations? In this paper, we address this problem for
permanent faults. We motivate the need for a fault tolerant MPI specification
by an in-depth consideration of recent innovations in DL algorithms and their
properties, which drive the need for specific fault tolerance features. We
present an in-depth discussion on the suitability of different parallelism
types (model, data and hybrid); a need (or lack thereof) for check-pointing of
any critical data structures; and most importantly, consideration for several
fault tolerance proposals (user-level fault mitigation (ULFM), Reinit) in MPI
and their applicability to fault tolerant DL implementations. We leverage a
distributed memory implementation of Caffe, currently available under the
Machine Learning Toolkit for Extreme Scale (MaTEx). We implement our approaches
by ex- tending MaTEx-Caffe for using ULFM-based implementation. Our evaluation
using the ImageNet dataset and AlexNet, and GoogLeNet neural network topologies
demonstrates the effectiveness of the proposed fault tolerant DL implementation
using OpenMPI based ULFM
Verification of the Tree-Based Hierarchical Read-Copy Update in the Linux Kernel
Read-Copy Update (RCU) is a scalable, high-performance Linux-kernel
synchronization mechanism that runs low-overhead readers concurrently with
updaters. Production-quality RCU implementations for multi-core systems are
decidedly non-trivial. Giving the ubiquity of Linux, a rare "million-year" bug
can occur several times per day across the installed base. Stringent validation
of RCU's complex behaviors is thus critically important. Exhaustive testing is
infeasible due to the exponential number of possible executions, which suggests
use of formal verification.
Previous verification efforts on RCU either focus on simple implementations
or use modeling languages, the latter requiring error-prone manual translation
that must be repeated frequently due to regular changes in the Linux kernel's
RCU implementation. In this paper, we first describe the implementation of Tree
RCU in the Linux kernel. We then discuss how to construct a model directly from
Tree RCU's source code in C, and use the CBMC model checker to verify its
safety and liveness properties. To our best knowledge, this is the first
verification of a significant part of RCU's source code, and is an important
step towards integration of formal verification into the Linux kernel's
regression test suite.Comment: This is a long version of a conference paper published in the 2018
Design, Automation and Test in Europe Conference (DATE
libcppa - Designing an Actor Semantic for C++11
Parallel hardware makes concurrency mandatory for efficient program
execution. However, writing concurrent software is both challenging and
error-prone. C++11 provides standard facilities for multiprogramming, such as
atomic operations with acquire/release semantics and RAII mutex locking, but
these primitives remain too low-level. Using them both correctly and
efficiently still requires expert knowledge and hand-crafting. The actor model
replaces implicit communication by sharing with an explicit message passing
mechanism. It applies to concurrency as well as distribution, and a lightweight
actor model implementation that schedules all actors in a properly
pre-dimensioned thread pool can outperform equivalent thread-based
applications. However, the actor model did not enter the domain of native
programming languages yet besides vendor-specific island solutions. With the
open source library libcppa, we want to combine the ability to build reliable
and distributed systems provided by the actor model with the performance and
resource-efficiency of C++11.Comment: 10 page
- …