HPC Cloud for Scientific and Business Applications: Taxonomy, Vision, and Research Challenges

High Performance Computing (HPC) clouds are becoming an alternative to on-premise clusters for executing scientific applications and business analytics services. Most research efforts in HPC cloud aim to understand the cost-benefit of moving resource-intensive applications from on-premise environments to public cloud platforms. Industry trends show hybrid environments are the natural path to get the best of the on-premise and cloud resources---steady (and sensitive) workloads can run on on-premise resources and peak demand can leverage remote resources in a pay-as-you-go manner. Nevertheless, there are plenty of questions to be answered in HPC cloud, which range from how to extract the best performance of an unknown underlying platform to what services are essential to make its usage easier. Moreover, the discussion on the right pricing and contractual models to fit small and large users is relevant for the sustainability of HPC clouds. This paper brings a survey and taxonomy of efforts in HPC cloud and a vision on what we believe is ahead of us, including a set of research challenges that, once tackled, can help advance businesses and scientific discoveries. This becomes particularly relevant due to the fast increasing wave of new HPC applications coming from big data and artificial intelligence.Comment: 29 pages, 5 figures, Published in ACM Computing Surveys (CSUR

Condor services for the Global Grid:interoperability between Condor and OGSA

In order for existing grid middleware to remain viable it is important to investigate their potentialfor integration with emerging grid standards and architectural schemes. The Open Grid ServicesArchitecture (OGSA), developed by the Globus Alliance and based on standard XML-based webservices technology, was the first attempt to identify the architectural components required tomigrate towards standardized global grid service delivery. This paper presents an investigation intothe integration of Condor, a widely adopted and sophisticated high-throughput computing softwarepackage, and OGSA; with the aim of bringing Condor in line with advances in Grid computing andprovide the Grid community with a mature suite of high-throughput computing job and resourcemanagement services. This report identifies mappings between elements of the OGSA and Condorinfrastructures, potential areas of conflict, and defines a set of complementary architectural optionsby which individual Condor services can be exposed as OGSA Grid services, in order to achieve aseamless integration of Condor resources in a standardized grid environment

Survey and Analysis of Production Distributed Computing Infrastructures

This report has two objectives. First, we describe a set of the production distributed infrastructures currently available, so that the reader has a basic understanding of them. This includes explaining why each infrastructure was created and made available and how it has succeeded and failed. The set is not complete, but we believe it is representative. Second, we describe the infrastructures in terms of their use, which is a combination of how they were designed to be used and how users have found ways to use them. Applications are often designed and created with specific infrastructures in mind, with both an appreciation of the existing capabilities provided by those infrastructures and an anticipation of their future capabilities. Here, the infrastructures we discuss were often designed and created with specific applications in mind, or at least specific types of applications. The reader should understand how the interplay between the infrastructure providers and the users leads to such usages, which we call usage modalities. These usage modalities are really abstractions that exist between the infrastructures and the applications; they influence the infrastructures by representing the applications, and they influence the ap- plications by representing the infrastructures

Enabling Connectors in Hierarchical Component Models

The continual growth of computing and storage capabilities enables scientific numerical applications to integrate more and more phenomena in their computations at the price of increased complexity. Hierarchical component models appear as an interesting approach to handle such complexity. However defining and implementing efficient interactions between hierarchical components is a difficult task, especially in the case of parallel and distributed applications. Connectors originating from Architecture Description Languages (ADL) offer a promising solution to this problem. There are however some cases where a simple combination of hierarchy and connectors in a single component model forces users to choose between an efficient implementation of components and their black box behavior. This paper describes HLCM, a model with connectors and hierarchy that provides /open connections/ as a mechanism to describe component interface that enhances encapsulation and eases component implementation replacement while supporting efficient interactions. Complex interactions such as data sharing and parallel method calls are successfully supported by HLCM. An implementation, based on model transformation and on CCM, illustrates its feasibility and benefits.La croissance continue des capacités de calcul et de stockage permet aux applications numériques d'intégrer un nombre croissant de phénomènes dans leurs calculs au prix d'une complexité accrue. Les modèles de composants hiérarchiques apparaissent comme une approche intéressante pour gérer cette complexité. Cependant, définir et implémenter des interactions efficaces entre composants hiérarchiques est une tâche difficile, d'autant plus dans le cas d'applications parallèles et distribuées. Les connecteurs issus des langages de description d'architecture (ADL) offrent une solution prometteuse à ce problème. Il y a cependant des cas où une simple combinaison de la hiérarchie et des connecteurs dans un modèle de composants unique oblige l'utilisateur à faire un choix entre des mises en œuvres efficaces pour les composants et leur comportement «boîte noire». Ce papier décrit HLCM, un modèle avec connecteurs et hiérarchie qui fournit le concept de /connexions ouvertes/ pour decrire les interfaces de composants. Ce méchanisme améliore l'encapsulation et facilite le remplacement des mises en œuvre de composant tout en permettant des interactions efficaces. Des interactions complexes telles que le partage de données ou les invocations de méthodes parallèles sont gérées avec succès par HLCM. Une mise en œuvre basée sur une transformation de modèle et sur CCM est utilisée pour illustrer sa faisabilité et ses bénéfices

A Grid-based solution for management and analysis of microarrays in distributed experiments

Several systems have been presented in the last years in order to manage the complexity of large microarray experiments. Although good results have been achieved, most systems tend to lack in one or more fields. A Grid based approach may provide a shared, standardized and reliable solution for storage and analysis of biological data, in order to maximize the results of experimental efforts. A Grid framework has been therefore adopted due to the necessity of remotely accessing large amounts of distributed data as well as to scale computational performances for terabyte datasets. Two different biological studies have been planned in order to highlight the benefits that can emerge from our Grid based platform. The described environment relies on storage services and computational services provided by the gLite Grid middleware. The Grid environment is also able to exploit the added value of metadata in order to let users better classify and search experiments. A state-of-art Grid portal has been implemented in order to hide the complexity of framework from end users and to make them able to easily access available services and data. The functional architecture of the portal is described. As a first test of the system performances, a gene expression analysis has been performed on a dataset of Affymetrix GeneChip® Rat Expression Array RAE230A, from the ArrayExpress database. The sequence of analysis includes three steps: (i) group opening and image set uploading, (ii) normalization, and (iii) model based gene expression (based on PM/MM difference model). Two different Linux versions (sequential and parallel) of the dChip software have been developed to implement the analysis and have been tested on a cluster. From results, it emerges that the parallelization of the analysis process and the execution of parallel jobs on distributed computational resources actually improve the performances. Moreover, the Grid environment have been tested both against the possibility of uploading and accessing distributed datasets through the Grid middleware and against its ability in managing the execution of jobs on distributed computational resources. Results from the Grid test will be discussed in a further paper