68,093 research outputs found
Checkpointing as a Service in Heterogeneous Cloud Environments
A non-invasive, cloud-agnostic approach is demonstrated for extending
existing cloud platforms to include checkpoint-restart capability. Most cloud
platforms currently rely on each application to provide its own fault
tolerance. A uniform mechanism within the cloud itself serves two purposes: (a)
direct support for long-running jobs, which would otherwise require a custom
fault-tolerant mechanism for each application; and (b) the administrative
capability to manage an over-subscribed cloud by temporarily swapping out jobs
when higher priority jobs arrive. An advantage of this uniform approach is that
it also supports parallel and distributed computations, over both TCP and
InfiniBand, thus allowing traditional HPC applications to take advantage of an
existing cloud infrastructure. Additionally, an integrated health-monitoring
mechanism detects when long-running jobs either fail or incur exceptionally low
performance, perhaps due to resource starvation, and proactively suspends the
job. The cloud-agnostic feature is demonstrated by applying the implementation
to two very different cloud platforms: Snooze and OpenStack. The use of a
cloud-agnostic architecture also enables, for the first time, migration of
applications from one cloud platform to another.Comment: 20 pages, 11 figures, appears in CCGrid, 201
TensorLayer: A Versatile Library for Efficient Deep Learning Development
Deep learning has enabled major advances in the fields of computer vision,
natural language processing, and multimedia among many others. Developing a
deep learning system is arduous and complex, as it involves constructing neural
network architectures, managing training/trained models, tuning optimization
process, preprocessing and organizing data, etc. TensorLayer is a versatile
Python library that aims at helping researchers and engineers efficiently
develop deep learning systems. It offers rich abstractions for neural networks,
model and data management, and parallel workflow mechanism. While boosting
efficiency, TensorLayer maintains both performance and scalability. TensorLayer
was released in September 2016 on GitHub, and has helped people from academia
and industry develop real-world applications of deep learning.Comment: ACM Multimedia 201
Let's Annotate to Let Our Code Run in Parallel
This paper presents an approach that exploits Java annotations to provide
meta information needed to automatically transform plain Java programs into
parallel code that can be run on multicore workstation. Programmers just need
to decorate the methods that will eventually be executed in parallel with
standard Java annotations. Annotations are automatically processed at
launch-time and parallel byte code is derived. Once in execution the program
automatically retrieves the information about the executing platform and
evaluates the information specified inside the annotations to transform the
byte-code into a semantically equivalent multithreaded version, depending on
the target architecture features. The results returned by the annotated
methods, when invoked, are futures with a wait-by-necessity semantics.Comment: 4 pages, 1 figur
HP-CERTI: Towards a high performance, high availability open source RTI for composable simulations (04F-SIW-014)
Composing simulations of complex systems from already existing simulation components remains a challenging issue. Motivations for composable simulation include generation of a given federation driven by operational requirements provided "on the fly". The High Level Architecture, initially developed for designing fully distributed simulations, can be considered as an interoperability standard for composing simulations from existing components. Requirements for constructing such complex simulations are quite different from those discussed for distributed simulations. Although interoperability and reusability remain essential, both high performance and availability have also to be considered to fulfill the requirements of the end user. ONERA is currently designing a High Performance / High Availability HLA Run-time Infrastructure from its open source implementation of HLA 1.3 specifications. HP-CERTI is a software package including two main components: the first one, SHM-CERTI, provides an optimized version of CERTI based on a shared memory communication scheme; the second one, Kerrighed-CERTI, allows the deployment of CERTI through the control of the Kerrighed Single System Image operating system for clusters, currently designed by IRISA. This paper describes the design of both high performance and availability Runtime Infrastructures, focusing on the architecture of SHM-CERTI. This work is carried out in the context of the COCA (High Performance Distributed Simulation and Models Reuse) Project, sponsored by the DGA/STTC (Délégation Générale pour l'Armement/Service des Stratégies Techniques et des Technologies Communes) of the French Ministry of Defense
Transparent Orchestration of Task-based Parallel Applications in Containers Platforms
This paper presents a framework to easily build and execute parallel applications in container-based distributed computing platforms in a user-transparent way. The proposed framework is a combination of the COMP Superscalar (COMPSs) programming model and runtime, which provides a straightforward way to develop task-based parallel applications from sequential codes, and containers management platforms that ease the deployment of applications in computing environments (as Docker, Mesos or Singularity). This framework provides scientists and developers with an easy way to implement parallel distributed applications and deploy them in a one-click fashion. We have built a prototype which integrates COMPSs with different containers engines in different scenarios: i) a Docker cluster, ii) a Mesos cluster, and iii) Singularity in an HPC cluster. We have evaluated the overhead in the building phase, deployment and execution of two benchmark applications compared to a Cloud testbed based on KVM and OpenStack and to the usage of bare metal nodes. We have observed an important gain in comparison to cloud environments during the building and deployment phases. This enables better adaptation of resources with respect to the computational load. In contrast, we detected an extra overhead during the execution, which is mainly due to the multi-host Docker networking.This work is partly supported by the Spanish Government through Programa Severo Ochoa (SEV-2015-0493), by the Spanish Ministry of Science and Technology through TIN2015-65316 project, by the Generalitat de Catalunya under contracts 2014-SGR-1051 and 2014-SGR-1272, and by the European Union through the Horizon 2020 research and innovation program under grant 690116 (EUBra-BIGSEA Project). Results presented in this paper were obtained using the Chameleon testbed supported by the National Science Foundation.Peer ReviewedPostprint (author's final draft
IMP Science Gateway: from the Portal to the Hub of Virtual Experimental Labs in Materials Science
"Science gateway" (SG) ideology means a user-friendly intuitive interface
between scientists (or scientific communities) and different software
components + various distributed computing infrastructures (DCIs) (like grids,
clouds, clusters), where researchers can focus on their scientific goals and
less on peculiarities of software/DCI. "IMP Science Gateway Portal"
(http://scigate.imp.kiev.ua) for complex workflow management and integration of
distributed computing resources (like clusters, service grids, desktop grids,
clouds) is presented. It is created on the basis of WS-PGRADE and gUSE
technologies, where WS-PGRADE is designed for science workflow operation and
gUSE - for smooth integration of available resources for parallel and
distributed computing in various heterogeneous distributed computing
infrastructures (DCI). The typical scientific workflows with possible scenarios
of its preparation and usage are presented. Several typical use cases for these
science applications (scientific workflows) are considered for molecular
dynamics (MD) simulations of complex behavior of various nanostructures
(nanoindentation of graphene layers, defect system relaxation in metal
nanocrystals, thermal stability of boron nitride nanotubes, etc.). The user
experience is analyzed in the context of its practical applications for MD
simulations in materials science, physics and nanotechnologies with available
heterogeneous DCIs. In conclusion, the "science gateway" approach - workflow
manager (like WS-PGRADE) + DCI resources manager (like gUSE)- gives opportunity
to use the SG portal (like "IMP Science Gateway Portal") in a very promising
way, namely, as a hub of various virtual experimental labs (different software
components + various requirements to resources) in the context of its practical
MD applications in materials science, physics, chemistry, biology, and
nanotechnologies.Comment: 6 pages, 5 figures, 3 tables; 6th International Workshop on Science
Gateways, IWSG-2014 (Dublin, Ireland, 3-5 June, 2014). arXiv admin note:
substantial text overlap with arXiv:1404.545
- âŠ