A job response time prediction method for production Grid computing environments

A major obstacle to the widespread adoption of Grid Computing in both the scientific community and industry sector is the difficulty of knowing in advance a job submission running cost that can be used to plan a correct allocation of resources. Traditional distributed computing solutions take advantage of homogeneous and open environments to propose prediction methods that use a detailed analysis of the hardware and software components. However, production Grid computing environments, which are large and use a complex and dynamic set of resources, present a different challenge. In Grid computing the source code of applications, programme libraries, and third-party software are not always available. In addition, Grid security policies may not agree to run hardware or software analysis tools to generate Grid components models. The objective of this research is the prediction of a job response time in production Grid computing environments. The solution is inspired by the concept of predicting future Grid behaviours based on previous experiences learned from heterogeneous Grid workload trace data. The research objective was selected with the aim of improving the Grid resource usability and the administration of Grid environments. The predicted data can be used to allocate resources in advance and inform forecasted finishing time and running costs before submission. The proposed Grid Computing Response Time Prediction (GRTP) method implements several internal stages where the workload traces are mined to produce a response time prediction for a given job. In addition, the GRTP method assesses the predicted result against the actual target job’s response time to inference information that is used to tune the methods setting parameters. The GRTP method was implemented and tested using a cross-validation technique to assess how the proposed solution generalises to independent data sets. The training set was taken from the Grid environment DAS (Distributed ASCI Supercomputer). The two testing sets were taken from AuverGrid and Grid5000 Grid environments Three consecutive tests assuming stable jobs, unstable jobs, and using a job type method to select the most appropriate prediction function were carried out. The tests offered a significant increase in prediction performance for data mining based methods applied in Grid computing environments. For instance, in Grid5000 the GRTP method answered 77 percent of job prediction requests with an error of less than 10 percent. While in the same environment, the most effective and accurate method using workload traces was only able to predict 32 percent of the cases within the same range of error. The GRTP method was able to handle unexpected changes in resources and services which affect the job response time trends and was able to adapt to new scenarios. The tests showed that the proposed GRTP method is capable of predicting job response time requests and it also improves the prediction quality when compared to other current solutions

gLite sur Grid'5000: vers une plate-forme d'expérimentation à taille réelle pour les grilles de production

National audienceThe Grid has become a huge and important instrument, playing a key role in the everyday work of many researchers. A large amount of software is being developed both to manage the grid infrastructure itself (gLite middleware), to facilitate the task of grid users (e.g workflow managers, pilot job managers, etc.), and to run the computations. That software must be designed to handle network and services outages in a highly distributed environment, while still providing the expected performance. It is inconvenient to test software using the production infrastructure, since (1) it might not exhibit the behaviour that is required to test extreme conditions (services are unlikely to crash as often as required when testing fault tolerance); (2) it might not be possible to replace key parts of the infrastructure without degrading the user experience; (3) experiments are not easily reproduced in production conditions. In this paper, we present our ongoing work on deploying the gLite middleware on the Grid'5000 testbed, a scientific instrument designed to support research on parallel, large-scale and distributed computing. Tools were written to automatize the deployment of the gLite middleware on several sites and clusters, resulting in a deployment of 5 sites and 8 clusters on 441 machines in less than an hour. This provides a solid basis for future experiments on the gLite middleware and on software that interact with the middleware

An OGSA Middleware for Managing Medical Images Using Ontologies

The final publication is available at Springer via http://dx.doi.org/ 10.1007/s10877-005-0675-0This article presents a Middleware based on Grid Technologies that addresses the problem of sharing, transferring and processing DICOM medical images in a distributed environment using an ontological schema to create virtual communities and to define common targets. It defines a distributed storage that builds-up virtual repositories integrating different individual image repositories providing global searching, progressive transmission, automatic encryption and pseudo-anonimisation and a link to remote processing services. Users from a Virtual Organisation can share the cases that are relevant for their communities or research areas, epidemiological studies or even deeper analysis of complex individual cases. Software architecture has been defined for solving the problems that has been exposed before. Briefly, the architecture comprises five layers (from the more physical layer to the more logical layer) based in Grid Thecnologies. The lowest level layers (Core Middleware Layer and Server Services layer) are composed of Grid Services that implement the global managing of resources. The Middleware Components Layer provides a transparent view of the Grid environment and it has been the main objective of this work. Finally, the upest layer (the Application Layer) comprises the applications, and a simple application has been implemented for testing the components developed in the Components Middleware Layer. Other side-results of this work are the services developed in the Middleware Components Layer for managing DICOM images, creating virtual DICOM storages, progressive transmission, automatic encryption and pseudo-anonimisation depending on the ontologies. Other results, such as the Grid Services developed in the lowest layers, are also described in this article. Finally a brief performance analysis and several snapshots from the applications developed are shown. The performance analysis proves that the components developed in this work provide image processing applications with new possibilities for large-scale sharing, management and processing of DICOM images. The results show that the components fulfil the objectives proposed. The extensibility of the system is achieved by the use of open methods and protocols, so new components can be easily added.Blanquer Espert, I.; Hernández García, V.; Segrelles Quilis, JD. (2005). An OGSA Middleware for Managing Medical Images Using Ontologies. Journal of Clinical Monitoring and Computing. 19:295-305. doi:10.1007/s10877-005-0675-0S29530519“European DataGrid Project”. http://www.eu-datagrid.org.“Biomedical Informatics Research”. http://www.nbirn.net/.“ACI project MEDIGRID: medical data storage and processing on the GRID”.http://www.creatis.insa-lyon.fr/MEDIGRID/.“Information eXtraction from Images (IXI) Grid Services for Medical Imaging”. Working Notes of the Workshop on Distributed Databases and processing in Medical Image Computing (DIDAMIC'04). Pag 65.“NeuroBase: Management of Distributed and Heterogeneous Information Sources in Neuroimaging”. Working Notes of the Workshop on Distributed Databases and processing in Medical Image Computing (DIDAMIC'04). Pag 85.Digital Imaging and Communications in Medicine (DICOM) Part 10: Media Storage and File Format for Media Interchange. National Electrical Manufacturers Association, 1300 N. 17th Street, Rosslyn, Virginia 22209 USA.“Open Grid Services Architecture (OGSA)”, http://www.globus.org/ogsa.Globus alliance Home Page. “Relevant documents”, http://www.globus.orgAllen Wyke R, Watt A, “XML Schema Essentials”. Wiley Computer Pub. ISBN 0-471-412597Web security and commerce/Simson Garfinkel. - Cambridge: O'Reilly, 1997. - 483 p.; 23 cm. ISBN 1565922697“The GridFTP Protocol and Software”. http://www-fp.globus.org/datagrid/gridftp.html.JPEG2000: Image compression fundamentals, standards and practice/David S. Taubman, Michael W. Marcellin. – Boston [etc.] : Kluwer Academic, cop. 2002. - XIX, 773 p.; 24 cm. + 1 CD-Rom - (The Kluwer international series in engineering and computer science) ISBN 079237519XBradley J, Erickson MD, “Irreversible Compression of Medical Images”, Dpt. Radiology, Mayo F., Rochester, MN, Jo. of D. Imaging, DOI: 10.1007/s10278-002-0001-z, 02.Monitoring & Discovery System (MDS)” http://www-unix.globus.org/toolkit/mds/“Key management for encrypted data storage in distributed systems”. Proceedings of HeathGrid 2004

Experimental Study of Remote Job Submission and Execution on LRM through Grid Computing Mechanisms

Remote job submission and execution is fundamental requirement of distributed computing done using Cluster computing. However, Cluster computing limits usage within a single organization. Grid computing environment can allow use of resources for remote job execution that are available in other organizations. This paper discusses concepts of batch-job execution using LRM and using Grid. The paper discusses two ways of preparing test Grid computing environment that we use for experimental testing of concepts. This paper presents experimental testing of remote job submission and execution mechanisms through LRM specific way and Grid computing ways. Moreover, the paper also discusses various problems faced while working with Grid computing environment and discusses their trouble-shootings. The understanding and experimental testing presented in this paper would become very useful to researchers who are new to the field of job management in Grid.Comment: Fourth International Conference on Advanced Computing & Communication Technologies (ACCT), 201

Towards Grid Interoperability

The Grid paradigm promises to provide global access to computing resources, data storage and experimental instruments. It also provides an elegant solution to many resource administration and provisioning problems while offering a platform for collaboration and resource sharing. Although substantial progress has been made towards these goals, nevertheless there is still a lot of work to be done until the Grid can deliver its promises. One of the central issues is the development of standards and Grid interoperability. Job execution is one of the key capabilities in all Grid environments. This is a well understood, mature area with standards and implementations. This paper describes some proof of concept experiments demonstrating the interoperability between various Grid environments

The OMII Software – Demonstrations and Comparisons between two different deployments for Client-Server Distributed Systems

This paper describes the key elements of the OMII software and the scenarios which OMII software can be deployed to achieve distributed computing in the UK e-Science Community, where two different deployments for Client-Server distributed systems are demonstrated. Scenarios and experiments for each deployment have been described, with its advantages and disadvantages compared and analyzed. We conclude that our first deployment is more relevant for system administrators or developers, and the second deployment is more suitable for users’ perspective which they can send and check job status for hundred job submissions

A DevOps approach to integration of software components in an EU research project

We present a description of the development and deployment infrastructure being created to support the integration effort of HARNESS, an EU FP7 project. HARNESS is a multi-partner research project intended to bring the power of heterogeneous resources to the cloud. It consists of a number of different services and technologies that interact with the OpenStack cloud computing platform at various levels. Many of these components are being developed independently by different teams at different locations across Europe, and keeping the work fully integrated is a challenge. We use a combination of Vagrant based virtual machines, Docker containers, and Ansible playbooks to provide a consistent and up-to-date environment to each developer. The same playbooks used to configure local virtual machines are also used to manage a static testbed with heterogeneous compute and storage devices, and to automate ephemeral larger-scale deployments to Grid5000. Access to internal projects is managed by GitLab, and automated testing of services within Docker-based environments and integrated deployments within virtual-machines is provided by Buildbot