Survey and Analysis of Production Distributed Computing Infrastructures

This report has two objectives. First, we describe a set of the production distributed infrastructures currently available, so that the reader has a basic understanding of them. This includes explaining why each infrastructure was created and made available and how it has succeeded and failed. The set is not complete, but we believe it is representative. Second, we describe the infrastructures in terms of their use, which is a combination of how they were designed to be used and how users have found ways to use them. Applications are often designed and created with specific infrastructures in mind, with both an appreciation of the existing capabilities provided by those infrastructures and an anticipation of their future capabilities. Here, the infrastructures we discuss were often designed and created with specific applications in mind, or at least specific types of applications. The reader should understand how the interplay between the infrastructure providers and the users leads to such usages, which we call usage modalities. These usage modalities are really abstractions that exist between the infrastructures and the applications; they influence the infrastructures by representing the applications, and they influence the ap- plications by representing the infrastructures

Report about the collaboration between UITS/Research Technologies at Indiana University and the Center for Information Services and High Performance Computing at Technische Universität Dresden, Germany (2011-2012)

This report lists the activities and outcomes for July 2011-June 2012 of the collaboration between Research Technologies, a division of University Information Technology Services at Indiana University (IU), and the Center for Information Services and High Performance Computing (ZIH) at Technische Universität Dresden.This material is based upon work supported in part by the National Science Foundation under Grant No. 0910812 to Indiana University for "FutureGrid: An Experimental, High-Performance Grid Test-bed." Partners in the FutureGrid project include San Diego Supercomputer Center at UC San Diego, University of Chicago, University of Florida, University of Southern California, University of Tennessee at Knoxville, University of Texas at Austin, Purdue University, University of Virginia, and T-U Dresden. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the NSF

Report about the collaboration between UITS/Research Technologies at Indiana University and the Center for Information Services and High Performance Computing at Technische Universität Dresden, Germany

This report lists the activities and outcomes of the collaboration between Research Technologies, a division of University Information Technology Services at Indiana University (IU), and the Center for Information Services and High Performance Computing (ZIH) at Technische Universität Dresden.This material is based upon work supported in part by the National Science Foundation under Grant No. 0910812 to Indiana University for "FutureGrid: An Experimental, High-Performance Grid Test-bed." Partners in the FutureGrid project include San Diego Supercomputer Center at UC San Diego, University of Chicago, University of Florida, University of Southern California, University of Tennessee at Knoxville, University of Texas at Austin, Purdue University, University of Virginia, and T-U Dresden. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the NSF

Adding Virtualization Capabilities to Grid'5000

Ce rapport révisé a fait l'objet d'une publication, voir hal-00946971Almost ten years after its premises, the Grid'5000 testbed has become one of the most complete testbed for designing or evaluating large-scale distributed systems. Initially dedicated to the study of High Performance Computing, the infrastructure has evolved to address wider concerns related to Desktop Computing, the Internet of Services and more recently the Cloud Computing paradigm. This report present recent improvements of the Grid'5000 software and services stack to support large-scale experiments using virtualization technologies as building blocks. Such contributions include the deployment of customized software environments, the reservation of dedicated network domain and the possibility to isolate them from the others, and the automation of experiments with a REST API. We illustrate the interest of these contributions by describing three different use-cases of large-scale experiments on the Grid'5000 testbed. The first one leverages virtual machines to conduct larger experiments spread over 4000 peers. The second one describes the deployment of 10000 KVM instances over 4 Grid'5000 sites. Finally, the last use case introduces a one-click deployment tool to easily deploy major IaaS solutions. The conclusion highlights some important challenges of Grid'5000 related to the use of OpenFlow and to the management of applications dealing with tremendous amount of data.Dix ans environ après ses prémisses, la plate-forme Grid'5000 est devenue une des plates-formes les plus complètes utilisée pour la conception et l'évaluation de systèmes distribués à grande échelle. Dédiée initialement au calcul à haute performance, l'infrastructure a évolué pour supporter un ensemble de problèmes plus vaste liés au calcul de type Desktop, l'internet des objets et plus récemment l'informatique dans les nuages (aussi appelé Cloud Computing). Ce rapport présente les améliorations récentes apportées au logiciels et pile de services pour supporter les expérimentations à grande échelle utilisant les technologies de virtualisation comme blocs de base. Nos contributions incluent le déploiement d'environnements logiciels customisés, la réservation de domaines réseaux dédiés et la possibilité de les isoler entre eux, et l'automatisation des expérimentations grâce à une API REST. Nous illustrons l'intérêt de ces contributions en décrivant trois expériences à large échelle sur la plate-forme Grid'5000. La première expérience utilise des machines virtuelles pour conduire des expérimentations de grande taille sur 4000 pairs. La seconde expérience décrit le déploiement de 10000 instances KVM sur 4 sites Grid'5000. Enfin le dernier exemple présente un outil de déploiement simple pour déployer des solutions de Cloud de type IaaS. La conclusion discute de prochains défis importants de Grid'5000 liés à l'utilisation d'OpenFlow et à la gestion d'applications gérant des grandes masses de données

Adding Virtualization Capabilities to Grid'5000

Ce rapport révisé a fait l'objet d'une publication, voir hal-00946971Almost ten years after its premises, the Grid'5000 testbed has become one of the most complete testbed for designing or evaluating large-scale distributed systems. Initially dedicated to the study of High Performance Computing, the infrastructure has evolved to address wider concerns related to Desktop Computing, the Internet of Services and more recently the Cloud Computing paradigm. This report present recent improvements of the Grid'5000 software and services stack to support large-scale experiments using virtualization technologies as building blocks. Such contributions include the deployment of customized software environments, the reservation of dedicated network domain and the possibility to isolate them from the others, and the automation of experiments with a REST API. We illustrate the interest of these contributions by describing three different use-cases of large-scale experiments on the Grid'5000 testbed. The first one leverages virtual machines to conduct larger experiments spread over 4000 peers. The second one describes the deployment of 10000 KVM instances over 4 Grid'5000 sites. Finally, the last use case introduces a one-click deployment tool to easily deploy major IaaS solutions. The conclusion highlights some important challenges of Grid'5000 related to the use of OpenFlow and to the management of applications dealing with tremendous amount of data.Dix ans environ après ses prémisses, la plate-forme Grid'5000 est devenue une des plates-formes les plus complètes utilisée pour la conception et l'évaluation de systèmes distribués à grande échelle. Dédiée initialement au calcul à haute performance, l'infrastructure a évolué pour supporter un ensemble de problèmes plus vaste liés au calcul de type Desktop, l'internet des objets et plus récemment l'informatique dans les nuages (aussi appelé Cloud Computing). Ce rapport présente les améliorations récentes apportées au logiciels et pile de services pour supporter les expérimentations à grande échelle utilisant les technologies de virtualisation comme blocs de base. Nos contributions incluent le déploiement d'environnements logiciels customisés, la réservation de domaines réseaux dédiés et la possibilité de les isoler entre eux, et l'automatisation des expérimentations grâce à une API REST. Nous illustrons l'intérêt de ces contributions en décrivant trois expériences à large échelle sur la plate-forme Grid'5000. La première expérience utilise des machines virtuelles pour conduire des expérimentations de grande taille sur 4000 pairs. La seconde expérience décrit le déploiement de 10000 instances KVM sur 4 sites Grid'5000. Enfin le dernier exemple présente un outil de déploiement simple pour déployer des solutions de Cloud de type IaaS. La conclusion discute de prochains défis importants de Grid'5000 liés à l'utilisation d'OpenFlow et à la gestion d'applications gérant des grandes masses de données

Reproducible Software Appliances for Experimentation

International audienceExperiment reproducibility is a milestone of the scientific method. Reproducibility of experiments in computer science would bring several advantages such as code re-usability and technology transfer. The reproducibility problem in computer science has been solved partially, addressing particular class of applications or single machine setups. In this paper we present our approach oriented to setup complex environments for experimentation, environments that require a lot of configuration and the installation of several software packages. The main objective of our approach is to enable the exact and independent reconstruction of a given software environment and the reuse of code. We present a simple and small software appliance generator that helps an experimenter to construct a specific software stack that can be deployed on different available testbeds

January 1 - December 31, 2012

This report summarizes training, education, and outreach activities for calendar 2012 of PTI and affiliated organizations, including the School of Informatics and Computing, Office of the Vice President for Information Technology, and Maurer School of Law. Reported activities include those led by PTI Research Centers (Center for Applied Cybersecurity Research, Center for Research in Extreme Scale Technologies, Data to Insight Center, Digital Science Center) and Service and Cyberinfrastructure Centers (Research Technologies Division of University Information Technology Services, National Center for Genome Assembly Support

Jetstream: A self-provisoned, scalable science and engineering cloud environment

The paper describes the motivation behind Jetstream, its functions, hardware configuration, software environment, user interface, design, use cases, relationships with other projects such as Wrangler and iPlant, and challenges in implementation.Funded by the National Science Foundation Award #ACI - 144560

Usage of UITS advanced research cyberinfrastructure for 2011

IU has a proud tradition in open access to its research computing and cyberinfrastructure (CI) facilities, going back to the precedents set by Marshall Wrubel (appointed the first permanent director of the IU Research Computing Center in 1955). Starting in 1997 President Myles Brand and then-Vice President Michael McRobbie initiated a tremendous acceleration in growth of IU’s cyberinfrastructure facilities through developing and then executing the first Indiana University Information Technology Strategic Plan. Through a decade and a half of purposeful execution of excellent strategies in support for research and scholarly activities generally, University Information Technology Services (UITS) has provided exceptional support to a group of researchers. This includes usage from disciplines that are among the traditional users of high performance computing – physics, chemistry, and astronomy, as well as emerging areas of application of HPC including biology, business, and the arts