Grid Data Management in Action: Experience in Running and Supporting Data Management Services in the EU DataGrid Project

In the first phase of the EU DataGrid (EDG) project, a Data Management System has been implemented and provided for deployment. The components of the current EDG Testbed are: a prototype of a Replica Manager Service built around the basic services provided by Globus, a centralised Replica Catalogue to store information about physical locations of files, and the Grid Data Mirroring Package (GDMP) that is widely used in various HEP collaborations in Europe and the US for data mirroring. During this year these services have been refined and made more robust so that they are fit to be used in a pre-production environment. Application users have been using this first release of the Data Management Services for more than a year. In the paper we present the components and their interaction, our implementation and experience as well as the feedback received from our user communities. We have resolved not only issues regarding integration with other EDG service components but also many of the interoperability issues with components of our partner projects in Europe and the U.S. The paper concludes with the basic lessons learned during this operation. These conclusions provide the motivation for the architecture of the next generation of Data Management Services that will be deployed in EDG during 2003.Comment: Talk from the 2003 Computing in High Energy and Nuclear Physics (CHEP03), La Jolla, Ca, USA, March 2003, 9 pages, LaTeX, PSN: TUAT007 all figures are in the directory "figures

ATLAS and CMS applications on the WorldGrid testbed

WorldGrid is an intercontinental testbed spanning Europe and the US integrating architecturally different Grid implementations based on the Globus toolkit. It has been developed in the context of the DataTAG and iVDGL projects, and successfully demonstrated during the WorldGrid demos at IST2002 (Copenhagen) and SC2002 (Baltimore). Two HEP experiments, ATLAS and CMS, successful exploited the WorldGrid testbed for executing jobs simulating the response of their detectors to physics eve nts produced by real collisions expected at the LHC accelerator starting from 2007. This data intensive activity has been run since many years on local dedicated computing farms consisting of hundreds of nodes and Terabytes of disk and tape storage. Within the WorldGrid testbed, for the first time HEP simulation jobs were submitted and run indifferently on US and European resources, despite of their underlying different Grid implementations, and produced data which could be retrieved and further analysed on the submitting machine, or simply stored on the remote resources and registered on a Replica Catalogue which made them available to the Grid for further processing. In this contribution we describe the job submission from Europe for both ATLAS and CMS applications, performed through the GENIUS portal operating on top of an EDG User Interface submitting to an EDG Resource Broker, pointing out the chosen interoperability solutions which made US and European resources equivalent from the applications point of view, the data management in the WorldGrid environment, and the CMS specific production tools which were interfaced to the GENIUS portal.Comment: Poster paper from the 2003 Computing in High Energy and Nuclear Physics (CHEP03), La Jolla, Ca, USA, March 2003, 10 pages, PDF. PSN TUCP004; added credit to funding agenc

Gridlab - a grid application toolkid and testbed

In this paper we present the new project called GridLab which is funded by the European Commission under the Fifth Framework Programme. The GridLab project, made up of computer scientists, astrophysicists and other scientists from various application areas, will develop and implement the grid application toolkit (GAT) together with a set of services to enable easy and efficient use of Grid resources in a real and production grid environment. GAT will provide core, easy to use functionality through a carefully constructed set of generic higher level grid APIs through which an application will be able to call the grid services laying beneath in order to perform efficiently in the Grid environment using various, dramatically wild application scenarios

Grid Information Technology as a New Technological Tool for e-Science, Healthcare and Life Science

Nowadays, scientific projects require collaborative environments and powerful computing resources capable of handling huge quantities of data, which gives rise to e-Science. These requirements are evident in the need to optimise time and efforts in activities to do with health. When e-Science focuses on the collaborative handling of all the information generated in clinical medicine and health, e-Health is the result. Scientists are taking increasing interest in an emerging technology – Grid Information Technology – that may offer a solution to their current needs. The current work aims to survey how e-Science is using this technology all around the world. We also argue that the technology may provide an ideal solution for the new challenges facing e-Health and Life Science.Hoy en día, los proyectos científicos requieren poderosos recursos de computación capaces de manejar grandes cantidades de datos, los cuales han dado paso a la ciencia electrónica (e-ciencia). Estos requerimientos se hacen evidentes en la necesidad de optimizar tiempo y esfuerzos en actividades relacionadas con la salud. Cuando la e-ciencia se enfoca en el manejo colaborativo de toda la información generada en la medicina clínica y la salud, da como resultado la salud electrónica (e-salud). Los científicos se han interesado cada vez más y más en una tecnología emergente, como lo es la Tecnología de información en red, la que puede ofrecer solución a sus necesidades cotidianas. El siguiente trabajo apunta a examinar como la e-ciencia es empleada en el mundo. También se discute que la tecnología puede proveer una solución ideal para encarar nuevos desafíos en e-salud y Ciencias de la Vida.Nowadays, scientific projects require collaborative environments and powerful computing resources capable of handling huge quantities of data, which gives rise to e-Science. These requirements are evident in the need to optimise time and efforts in activities to do with health. When e-Science focuses on the collaborative handling of all the information generated in clinical medicine and health, e-Health is the result. Scientists are taking increasing interest in an emerging technology – Grid Information Technology – that may offer a solution to their current needs. The current work aims to survey how e-Science is using this technology all around the world. We also argue that the technology may provide an ideal solution for the new challenges facing e-Health and Life Science

Multi-voxel fMRI analysis using an high throughput grid framework

Mestrado em Engenharia Biomédica - Instrumentação, Sinal e ImagemO presente trabalho apresenta uma nova abordagem à análise de imagens de RMf do cérebro, especificamente a utilização de medidas associativas na análise de séries temporais de RMf. Este tipo específico de análise, computacionalmente intensivo, requer recursos que normalmente não se encontram disponíveis em ambientes clínicos. Redes Grid é um novo paradigma de computação distribuída de elevada performance que pode ser utilizado para potenciar a utilização deste tipo de análise, disponibilizando a capacidade de computação necessária. Implementouse um framework que permite a utilização de uma infraestrutura Grid para correr este tipo de análise de forma transparente, viabilizando a sua utilização em ambientes clínicos, onde o tempo é um factor crítico. ABSTRACT: This work, introduces a new approach to fMRI brain image analysis, namely multivoxel fMRI association analysis. The problem associated with this type of approach is that requires a large computing capacity that is not normally available at clinical sites. To enable this specific type of analysis we are required to use High Performance Computing paradigms. In this context we analysed the use of Grid computing and implemented a framework that allows running the multivoxel fMRI association analysis using a grid infrastructure resources. The use of this framework makes this type of analysis usable in clinical environments where time constraints can have a vital importance

Grid accounting for computing and storage resources towards standardization

In the last years, we have seen a growing interest of the scientific community first and commercial vendors then, in new technologies like Grid and Cloud computing. The first in particular, was born to meet the enormous computational requests mostly coming from physic experiments, especially Large Hadron Collider's (LHC) experiments at Conseil Européen pour la Recherche Nucléaire (European Laboratory for Particle Physics) (CERN) in Geneva. Other scientific disciplines that are also benefiting from those technologies are biology, astronomy, earth sciences, life sciences, etc. Grid systems allow the sharing of heterogeneous computational and storage resources between different geographically distributed institutes, agencies or universities. For this purpose technologies have been developed to allow communication, authentication, storing and processing of the required software and scientific data. This allows different scientific communities the access to computational resources that a single institute could not host for logistical and cost reasons. Grid systems were not the only answer to this growing need of resources of different communities. At the same time, in the last years, we have seen the affirmation of the so called Cloud Computing. Cloud computing is a model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g.: networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction. The use of both computational paradigms and the utilization of storage resources, leverage on different authentication and authorization tools. The utilization of those technologies requires systems for the accounting of the consumed resources. Those systems are built on the top of the existing infrastructure and they collect all the needed data related to the users, groups and resources utilized. This information is then collected in central repositories where they can be analyzed and aggregated. Open Grid Forum (OGF) is the international organism that works to develop standards in the Grid environment. Usage Record - Working Group (UR-WG) is a group, born within OGF aiming at standardizing the Usage Record (UR) structure and publication for different kinds of resources. Up to now, the emphasis has been on the accounting for computational resources. With time it came out the need to expand those concepts to other aspects and especially to a definition and implementation of a standard UR for storage accounting. Several extensions to the UR definition are proposed in this thesis and the proposed developments in this field are described. The Distributed Grid Accounting System (DGAS) has been chosen, among other tools available, as the accounting system for the Italian Grid and is also adopted in other countries such as Greece and Germany. Together with HLRmon, it offers a complete accounting system and it is the tool that has been used during the writing of the thesis at INFN-CNAF. • In Chapter 1, I will focus on the paradigm of distributed computing and the Grid infrastructure will be introduced with particular emphasis on the gLite middleware and the EGI-InSPIRE project. • In Chapter 2, I will discuss some Grid accounting systems for computational resources with particular stress for DGAS. • In Chapter 3, the cross-check monitoring system used to check the correctness of the gathered data at the INFN-CNAF's Tier1 is presented. • In Chapter 4, another important aspect on accounting, accounting for storage resources, is introduced and the definition of a standard UR for storage accounting is presented. • In Chapter 5, an implementation of a new accounting system for the storage that uses the definitions given in Chapter 4 is presented. • In Chapter 6, the focus of the thesis move on the performance and reliability tests performed on the latest development release of DGAS that implements ActiveMQ as a standard transport mechanism. • In Appendix A are collected the BASH scripts and SQL code that are part of the cross-check tool described in Chapter 3. • In Appendix B are collected the scripts used in the implementation of the accounting system described in Chapter 5. • In Appendix C are collected the scripts and configurations used for the tests of the ActiveMQ implementation of DGAS described in Chapter 6. • In Appendix D are collected the publications in which I contributed during the thesis wor

Ethernet Networks for Real-Time Use in the ATLAS Experiment

Ethernet became today's de-facto standard technology for local area networks. Defined by the IEEE 802.3 and 802.1 working groups, the Ethernet standards cover technologies deployed at the first two layers of the OSI protocol stack. The architecture of modern Ethernet networks is based on switches. The switches are devices usually built using a store-and-forward concept. At the highest level, they can be seen as a collection of queues and mathematically modelled by means of queuing theory. However, the traffic profiles on modern Ethernet networks are rather different from those assumed in classical queuing theory. The standard recommendations for evaluating the performance of network devices define the values that should be measured but do not specify a way of reconciling these values with the internal architecture of the switches. The introduction of the 10 Gigabit Ethernet standard provided a direct gateway from the LAN to the WAN by the means of the WAN PHY. Certain aspects related to the actual use of WAN PHY technology were vaguely defined by the standard. The ATLAS experiment at CERN is scheduled to start operation at CERN in 2007. The communication infrastructure of the Trigger and Data Acquisition System will be built using Ethernet networks. The real-time operational needs impose a requirement for predictable performance on the network part. In view of the diversity of the architectures of Ethernet devices, testing and modelling is required in order to make sure the full system will operate predictably. This thesis focuses on the testing part of the problem and addresses issues in determining the performance for both LAN and WAN connections. The problem of reconciling results from measurements to architectural details of the switches will also be tackled. We developed a scalable traffic generator system based on commercial-off-the-shelf Gigabit Ethernet network interface cards. The generator was able to transmit traffic at the nominal Gigabit Ethernet line rate for all frame sizes specified in the Ethernet standard. The calculation of latency was performed with accuracy in the range of +/- 200 ns. We indicate how certain features of switch architectures may be identified through accurate throughput and latency values measured for specific traffic distributions. At this stage, we present a detailed analysis of Ethernet broadcast support in modern switches. We use a similar hands-on approach to address the problem of extending Ethernet networks over long distances. Based on the 1 Gbit/s traffic generator used in the LAN, we develop a methodology to characterise point-to-point connections over long distance networks. At higher speeds, a combination of commercial traffic generators and high-end servers is employed to determine the performance of the connection. We demonstrate that the new 10 Gigabit Ethernet technology can interoperate with the installed base of SONET/SDH equipment through a series of experiments on point-to-point circuits deployed over long-distance network infrastructure in a multi-operator domain. In this process, we provide a holistic view of the end-to-end performance of 10 Gigabit Ethernet WAN PHY connections through a sequence of measurements starting at the physical transmission layer and continuing up to the transport layer of the OSI protocol stack

Recovery-oriented software architecture for grid applications (ROSA-Grids)

Grids are distributed systems that dynamically coordinate a large number of heterogeneous resources to execute large-scale projects. Examples of grid resources include high-performance computers, massive data stores, high bandwidth networking, telescopes, and synchrotrons. Failure in grids is arguably inevitable due to the massive scale and the heterogeneity of grid resources, the distribution of these resources over unreliable networks, the complexity of mechanisms that are needed to integrate such resources into a seamless utility, and the dynamic nature of the grid infrastructure that allows continuous changes to happen. To make matters worse, grid applications are generally long running, and these runs repeatedly require coordinated use of many resources at the same time. In this thesis, we propose the Recovery-Aware Components (RAC) approach. The RAC approach enables a grid application to handle failure reactively and proactively at the level of the smallest and independent execution unit of the application. The approach also combines runtime prediction with a proactive fault tolerance strategy. The RAC approach aims at improving the reliability of the grid application with the least overhead possible. Moreover, to allow a grid fault tolerance manager fine-tuned control and trading off of reliability gained and overhead paid, this thesis offers an architecture-aware modelling and simulation of reliability and overhead. The thesis demonstrates for a few of a dozen or so classes of application architecture already identified in prior research, that the typical architectural structure of the class can be captured in a few parameters. The work shows that these parameters suffice to achieve significant insight into, and control of, such tradeoffs. The contributions of our research project are as follows. We defined the RAC approach. We showed the usage of the RAC approach for improving the reliability of MapReduce and Combinational Logic grid applications. We provided Markov models that represent the execution behaviour of these applications for reliability and overhead analyses. We analysed the sensitivity of the reliability-overhead tradeoff of the RAC approach to the type of fault tolerance strategy, the parameters of a fault tolerance strategy, prediction interval and a predictor’s accuracy. The final contribution of our research is an experiment testbed that enables a grid fault tolerance expert to evaluate diverse fault tolerance support configurations, and then choose the one that will satisfy the reliability and cost requirements

Science 2.0 : sharing scientific data on the Grid

Mestrado em Engenharia de Computadores e TelemáticaA computação assumese cada vez mais como um recurso essencial em ciência, levando ao surgimento do termo eCiência para designar a utilização de tecnologias de computação avançada para suportar a realização de experiências científicas, a preservação e partilha do conhecimento. Uma das áreas de aplicação do conceito de eCiência é o tratamento e análise de imagens médicas. Os processos que lidam com imagem médica, tanto ao nível clínico como de investigação, são exigentes em relação ao suporte computacional, devido aos algoritmos de processamento de imagem que requerem e à elevada capacidade de armazenamento relacionada com volume das imagens geradas. As políticas públicas e os avanços tecnológicos recentes orientados para a eCiência, têm vindo a apoiar o desenvolvimento da computação em Grid, tanto a nível dos middlewares como da instalação de capacidade de produção, como um sistema de computação avançado que permite a partilha de recursos, instrumentos científicos e boas práticas em comunidades virtuais. Este trabalho tem como objectivo desenvolver uma estratégia e um protótipo para o armazenamento de dados médicos na Grid, visando a sua utilização em investigação. Uma preocupação diferenciadora prendese com o objectivo de colocar as potencialidades da Grid ao serviço de utilizadores não técnicos (e.g. médicos, investigadores), que acedem a serviços de processamento e de armazenamento e catalogação de dados de forma transparente, através de um portal Web. O protótipo desenvolvido permite a investigadores na área das neurociências, sem conhecimentos específicos da tecnologia Grid, armazenar imagens e analisálas em Grids de produção existentes, baseadas no middleware gLite. ABSTRACT: Computing has become an essential tool in modern science, leading to the appearance of the term eScience to designate the usage of advanced computing technologies to support the execution of scientific experiments, and the preservation and sharing of knowledge. One of eScience domain areas is the medical imaging analysis. The processes that deal with medical images, both at clinical and investigation level, are very demanding in terms of computational support, due to the analysis algorithms that involve large volumes of generated images, requiring high storage capabilities. The recent public policies and technological advances are eScience oriented, and have been supporting the development of Grid computing, both at the middleware level and at the installation of production capabilities in an advanced computing system, that allows the sharing of resources, scientific instrumentation and good practices among virtual communities. The main objective of this work is to develop a strategy and a prototype to allow the storage of medical data on the Grid, targeting a research environment. The differentiating concern of this work is the ability to provide the nonexperts users (e.g: doctors, researchers) access to the Grid services, like storage and processing, through a friendly Web interface. The developed prototype allows researchers in the field of neuroscience, without any specific knowledge of Grid technology, to store images and analyse them in production Grid infrastructures, based on the gLite middleware