26,874 research outputs found
What does fault tolerant Deep Learning need from MPI?
Deep Learning (DL) algorithms have become the de facto Machine Learning (ML)
algorithm for large scale data analysis. DL algorithms are computationally
expensive - even distributed DL implementations which use MPI require days of
training (model learning) time on commonly studied datasets. Long running DL
applications become susceptible to faults - requiring development of a fault
tolerant system infrastructure, in addition to fault tolerant DL algorithms.
This raises an important question: What is needed from MPI for de- signing
fault tolerant DL implementations? In this paper, we address this problem for
permanent faults. We motivate the need for a fault tolerant MPI specification
by an in-depth consideration of recent innovations in DL algorithms and their
properties, which drive the need for specific fault tolerance features. We
present an in-depth discussion on the suitability of different parallelism
types (model, data and hybrid); a need (or lack thereof) for check-pointing of
any critical data structures; and most importantly, consideration for several
fault tolerance proposals (user-level fault mitigation (ULFM), Reinit) in MPI
and their applicability to fault tolerant DL implementations. We leverage a
distributed memory implementation of Caffe, currently available under the
Machine Learning Toolkit for Extreme Scale (MaTEx). We implement our approaches
by ex- tending MaTEx-Caffe for using ULFM-based implementation. Our evaluation
using the ImageNet dataset and AlexNet, and GoogLeNet neural network topologies
demonstrates the effectiveness of the proposed fault tolerant DL implementation
using OpenMPI based ULFM
Analysis of RSVP-TE graceful restart
GMPLS is viewed as an attractive intelligent control plane for different network technologies and graceful restart is a key technique in ensuring this control plane is resilient and able to recover adequately from faults. This paper analyses the graceful restart mechanism proposed for a key GMPLS protocol, RSVP-TE. A novel analytical model, which may be readily adapted to study other protocols, is developed. This model allows the efficacy of graceful restart to be evaluated in a number of scenarios. It is found that, unsurprisingly, increasing control message loss and increasing the number of data plane connections both increased the time to complete recovery. It was also discovered that a threshold exists beyond which a relatively small change in the control message loss probability causes a disproportionately large increase in the time to complete recovery. The interesting findings in this work suggest that the performance of graceful restart is worthy of further investigation, with emphasis being placed on exploring procedures to optimise the performance of graceful restart
CRAFT: A library for easier application-level Checkpoint/Restart and Automatic Fault Tolerance
In order to efficiently use the future generations of supercomputers, fault
tolerance and power consumption are two of the prime challenges anticipated by
the High Performance Computing (HPC) community. Checkpoint/Restart (CR) has
been and still is the most widely used technique to deal with hard failures.
Application-level CR is the most effective CR technique in terms of overhead
efficiency but it takes a lot of implementation effort. This work presents the
implementation of our C++ based library CRAFT (Checkpoint-Restart and Automatic
Fault Tolerance), which serves two purposes. First, it provides an extendable
library that significantly eases the implementation of application-level
checkpointing. The most basic and frequently used checkpoint data types are
already part of CRAFT and can be directly used out of the box. The library can
be easily extended to add more data types. As means of overhead reduction, the
library offers a build-in asynchronous checkpointing mechanism and also
supports the Scalable Checkpoint/Restart (SCR) library for node level
checkpointing. Second, CRAFT provides an easier interface for User-Level
Failure Mitigation (ULFM) based dynamic process recovery, which significantly
reduces the complexity and effort of failure detection and communication
recovery mechanism. By utilizing both functionalities together, applications
can write application-level checkpoints and recover dynamically from process
failures with very limited programming effort. This work presents the design
and use of our library in detail. The associated overheads are thoroughly
analyzed using several benchmarks
Diffusion Component Analysis: Unraveling Functional Topology in Biological Networks
Complex biological systems have been successfully modeled by biochemical and
genetic interaction networks, typically gathered from high-throughput (HTP)
data. These networks can be used to infer functional relationships between
genes or proteins. Using the intuition that the topological role of a gene in a
network relates to its biological function, local or diffusion based
"guilt-by-association" and graph-theoretic methods have had success in
inferring gene functions. Here we seek to improve function prediction by
integrating diffusion-based methods with a novel dimensionality reduction
technique to overcome the incomplete and noisy nature of network data. In this
paper, we introduce diffusion component analysis (DCA), a framework that plugs
in a diffusion model and learns a low-dimensional vector representation of each
node to encode the topological properties of a network. As a proof of concept,
we demonstrate DCA's substantial improvement over state-of-the-art
diffusion-based approaches in predicting protein function from molecular
interaction networks. Moreover, our DCA framework can integrate multiple
networks from heterogeneous sources, consisting of genomic information,
biochemical experiments and other resources, to even further improve function
prediction. Yet another layer of performance gain is achieved by integrating
the DCA framework with support vector machines that take our node vector
representations as features. Overall, our DCA framework provides a novel
representation of nodes in a network that can be used as a plug-in architecture
to other machine learning algorithms to decipher topological properties of and
obtain novel insights into interactomes.Comment: RECOMB 201
Mirror, mirror on the wall, tell me, is the error small?
Do object part localization methods produce bilaterally symmetric results on
mirror images? Surprisingly not, even though state of the art methods augment
the training set with mirrored images. In this paper we take a closer look into
this issue. We first introduce the concept of mirrorability as the ability of a
model to produce symmetric results in mirrored images and introduce a
corresponding measure, namely the \textit{mirror error} that is defined as the
difference between the detection result on an image and the mirror of the
detection result on its mirror image. We evaluate the mirrorability of several
state of the art algorithms in two of the most intensively studied problems,
namely human pose estimation and face alignment. Our experiments lead to
several interesting findings: 1) Surprisingly, most of state of the art methods
struggle to preserve the mirror symmetry, despite the fact that they do have
very similar overall performance on the original and mirror images; 2) the low
mirrorability is not caused by training or testing sample bias - all algorithms
are trained on both the original images and their mirrored versions; 3) the
mirror error is strongly correlated to the localization/alignment error (with
correlation coefficients around 0.7). Since the mirror error is calculated
without knowledge of the ground truth, we show two interesting applications -
in the first it is used to guide the selection of difficult samples and in the
second to give feedback in a popular Cascaded Pose Regression method for face
alignment.Comment: 8 pages, 9 figure
- âŠ