This document reports on the six month survey of the UK e-Science demonstrator projects. The report details the eight project consultations in which we have engaged, and provides a summary of the likely compatibility of these projects with the Warwick middleware research. Two of these eight projects have been identified for further collaboration, details of which are documented. In addition, this consultation has been extended to include UK e-Science pilot projects and the details of planned collaboration with one of these pilot projects also forms part of this report

Jarvis, Stephen A.

Nudd, G. R.

English

Warwick Research Archives Portal Repository

http://wrap.warwick.ac.uk/            Original citation: Jarvis, Stephen A., 1970- and Nudd, G. R. (2004) Review of the e-science demonstrators and their compatibility with the Warwick middleware project. University of Warwick. Department of Computer Science. (Department of Computer Science Research report).  Permanent WRAP url: http://wrap.warwick.ac.uk/61311              Copyright and reuse: The Warwick Research Archive Portal (WRAP) makes this work by researchers of the University of Warwick available open access under the following conditions.  Copyright © and all moral rights to the version of the paper presented here belong to the individual author(s) and/or other copyright owners.  To the extent reasonable and practicable the material made available in WRAP has been checked for eligibility before being made available.  Copies of full items can be used for personal research or study, educational, or not-for-profit purposes without prior permission or charge.  Provided that the authors, title and full bibliographic details are credited, a hyperlink and/or URL is given for the original metadata page and the content is not changed in any way.  A note on versions: The version presented in WRAP is the published version or, version of record, and may be cited as it appears here.For more information, please contact the WRAP Team at: publications@warwick.ac.uk  $\VERJ,.\)' € '/,v4- m;&o_WATICKResearch Report 394Rrurw oF THE r-ScrrrucE DEMoNSTRAToRSAND THEIR COMPATIBILITY WITH THE WNRWICTMtoolewARE Pno.rrcrStephen Jaruis, Graham NuddDepartment of Computer ScienceUniversity of Warwick, Coventry CV4 7AL, UKWAryICKAuthors:Date:Review of the e-Science Demonstrators and theircompatibilify with the Warwick Middleware ProjectSAJarvis,GRNuddHigh Perfbnnance Systems Group,Department of Computer Science, University of Warwick31-3-2003Abstract:This document reports on the six month survey of the UK e-science demonstrator projects. Thereport details the eight project consultations in which we have engaged, and provides a summaryof the likely compatibility of these projects with the Warwick middleware research. Two of theseeight projects have been identified for further collaboration, details of which are documented. Inaddition, this consultation has been extended to include UK e-Science pilot projects and thedetails of planned collaboration with one of these pilot projects also forms part of this report.Reference: GR/S03058/01 RR394Performance-based Middleware for Grid ComputingIntroductionThe e-Science demonstrator projects provide a diverse collection of demonstrations andmethodologies that represent tasks which are considered pertinent to e-Science. These projectsare at various stages of completion - some have already ended and some are in the initial stagesas they rely on key infrastructural support which is currently being installed.The Warwick Middleware project aims to provide performance services for e-Science and e-Business applications. As part of the e-Science Open Call funding (GR/S03058/01), we haveagreed to identiff and then deliver results of the application of these middleware performanceservices to a chosen e-Science demonstrator application.This document reports on the six month survey of the UK e-Science demonstrator projects' Thereport details the eight project consultations in which we have engaged, and provides a summaryof the likely compatibility of these projects with the Warwick middleware research.To optimize research compatibility and cooperation within the UK e-Science programme, thissix-month study has also included additional consultation:The evaluation of UK e-science demonstrator projects: including the refinement of theseto one or a small subset of projects which will be the subject of more detailed scrutiny;An evaluation of a subset of UK e-Science pilot projects: to investigate whether themiddleware performance services might be compatible with some of the larger UK e-Science projects;An evaluation of the e-Business case studies in light of IBMs involvement in other UK e-Science work.ln order to assess the benefit and appropriateness of the application of the middleware to thedemonstrators, a number of broad suitability requirements have been applied:Will the Warwick middleware services bring added-value?Is the demonstrator (language, operating system etc.) compatible with the Warwickmiddleware tools?Is specialist equipment needed to run an image of the demonstrator at Warwick?Are resources available at the partner institution to support the coliaboration?This report contains: a list of the demonstrator consultations which have taken place during thepast six months; those demonstrator projects identified for further collaboration; a list of the pilotproject consultations which have taken place during the past six months; those pilot projectsidentified for further collaboration; a list of the supporting research which has been instigated toin order to underline this proposed collaboration.Demonstrator Project ConsultationsConsultations have taken place with eightmonths (a summary of these projects, thetheir location, are shown in Table 1).UK e-Science demonstrator projectsmain contacts with whom we haveduring the past sixbeen involved andProiect Contact Location3D OPT Microscopy Grid Richard BaldockNeil Chue HoneMRC Human Genetics Unit,EdinburghEPCC3D Mouse Grid Richard Baldock MRC Human Genetics UnitEdinburehChemical Structures(Comb-e-Chem)Mike Surridge University of Southampton(IT Innovation)Dynamic Brain Atlas(IXI)Derek Hill King's College LondonGODIVA Jochem MarotzkeJohn StarkDavid BovdsSouthampton OceanographyCentreRALGRAB (BDWorld) Alex Grav Cardiff UniversityNeSC ePortal Mark Parsons EPCCSAMD Keith CoieStephen PicklesManchester UniversityTable l: Overview of demonstrator project consultationsThis list of projects can be partitioned in to those for which performance services are notconsidered important (as the project has already finished, performance is not an importantcomponent of the demonstration etc.) - including GODIVA, NeSC ePortal and SAMD - thosewhich might benefit from the introduction of the performance services at a later date - includingComb-e-Chem and GODIVA - and those for which performance services are essential to thesuccess of the project - including Dynamic Brain Atlas and 3D OTP Microscopy Grid.Identified Demonstrator Projects for Further CollaborationIt is possible that projects such as Comb-e-Chem and GODIVA will demonstrate futurecompatibility with our middleware research, they are however not yet at the stage where we canstart to consider how this integration might take place.Detailed discussion and consultative visits have taken place between Warwick and EdinburghUniversity (and the MRC Edinburgh) with regard to the 3D OTP Microscopy Grid and alsoKing's College London with regard to the Dynamic Brain Atlas project. An advantage inselecting these two demonstrators for further collaboration is that they both have a considerableimage processing content. This area of expertise is supported by the Image Processing Group inthe Department of Computer Science at Warwick, and we are well placed therefore to be able tocontribute at the necessarv level ofdetail.3D OTP Microscopy GridThe 3D OPT demonstrator on the reconstruction of mouse embryo data is composed of threetasks - data capture, transfer and parallel reconstruction. Performance is critical to this project;for example, the reconstruction of the 3D image takes about 2-5 hours with a serial version of thealgorithm, which for the 3-channel image means an overnight run before the biologists canreview the results and reset the parameters as necessary.The basic algorithm takes a set of projection images, deconstructs them after some preprocessingto sets of sinograms, and then uses these during the reconstruction process; this is equivalent totaking an orthogonal cut through the data. The reconstruction process involves stacking thesections generated by the back-projection processing.A parallel version of this algorithm has been impiemented and demonstrated. This is currentlybeing re-engineered to run on an array of Sun netras at the MRC. The code is written in C andMPI and the Sun Grid Engine is used to handle scheduling. There are plans also to use Globus toconnect to remote resources. A stable version of this system is due to be complete by May 2003.It is likely that our middleware tools will directly benefit this image reconstruction process. Ourtools are currently engineered for C MPl-based code and it should be feasible to generateresource models for the hardware at the MRC.While our performance-based scheduling tools are currently based on a Globus/Condormiddleware stack (see below), we plan to investigate whether our performance co-scheduler canbe used alongside Sun's Grid Engine.Dynamic Brain AtlasAlthough the Dynamic Brain Atlas project has now finished, follow up research funding has beensought, and now granted, during the past six months. We offered partner suppoft in this bid andits success therefore means that there are now resources available at King's to be able to supportcontinued collaboration. The new project, known as IXI, started on the 17th February 2003.Two work areas have been started at Warwick to facilitate collaboration with this project. Thefirst involves additional support from Dr Chang-Tsun Li (from the Image Processing Group in theDepartment of Computer Science, Warwick) who is currently working on identif,ing core imageprocessing kernels which form part of the IXI project. The aim of this is to then test themiddleware tools on these image processing kemels, perform some in-house analysis to verify thebenefits which they offer, and then transfer these to the demonstrator later in the project.The second work area is the integration of the performance tools with Condor - the preferredintra-domain scheduler use in IXI. This has formed one of the major areas of research over thepast six months and has involved adapting our performance scheduler (known as Titan) so that itcan be used as a driver to Condor. By doing this the middleware tools should integrate into theIXI demonstrator piatform (and indeed any other that utilizes Condor at the intra-domain ievel).Progress has already been made on this work area, details of which can be found below.Pilot Project ConsultationsMany of the lessons learnt from the prototyping of the demonstrator projects have beentransferred to the development plattbrms of the larger pilot projects (e.g. SAMD andRealityGrid). Indeed the perceived importance of performance services is noticeably greater atthe pilot project level than at the demonstrator project level.This consultation has therefore been extended to include a subset of the UK e-Science pilotprojects, whose performance requirements are somewhat compatible (a summary of theseprojects, the main contacts with whom we have been involved and their location, are shown inTable 2).Proiect Contact LocationRealityGrid Stephen PicklesGraham RileyManchester UniversityEDiamond Mike BradyDavid GavaghanSteve HeisieOxford UniversityIBMDAME Karim Diemame Leeds UniversitlTable 2: Overview of pilot project consultationsThe analysis and development of techniques to address performance issues are clearly alreadyintegrated components of these pilot projects. It is important however to build on the researchexperience gained during the performance engineering of each of these pilot projects, and to usethis experience when considering the design and deployment of our own middleware tools.Identified Pilot Projects for Further CollaborationAs part of this process we have been involved in collaborative meetings with the GridInfrastructure team on the eDiamond project. Through Donna Dillenberger and Steve Heisig(IBM Watson) we are looking to share experience in two specific areas:How we define appiication/request topologies that describe the link between applicationcomponents. There are two aspects to this work, the language in which these application unitsshould be described (cunently the Open Group's ARM API) and then how these work unitscan be used during the performance analysis of applications.We are also looking into how user-level service provision can be mapped to these lower-levelwork units, particularly when these work units are processed across a distributed architecture.This links with the hWLM and eWLM projects that Donna Dillenberger and Steve Heisig areinvolved in at IBM and also the demonstrator for this work eDiamond.Experience in this areas is to be shared through conference call meetings and also throughmeetings at the Oxford e-Science regional centre (organized through David Gavaghan).Supporting ResearchSupporting image processing researchAlthough our research is to be tested in the context of an e-Science demonstrator project, we aretrying to avoid building dependencies that are likely to slow our work down. We have done thisby identifying three possible e-Science demonstrators that have similar image processingcharacteristics (3D OPT Microscopy Grid, IXi and eDiamond). One of our academics from theImage Processing Research Group at Warwick (Dr Chang-Tsun Li) is currently working onidentifying core kernel characteristics of these projects, the idea being that if we can test andverify our tools on these core kemels then the integration with the demonstrator/s should be madeeasier later in the project.For example, the image segmentation techniques that enable low-level image analysis in projectssuch as the Dynamic Brain Atlas (IXi), have been the subject of detailed analysis at Warwick,see:CT Li, Multiresolution Image Segmentation Integrating Gibbs Sampler and Region MergingAlgorithm, Signal Processing, 83(1):67-78, 2003.R Wilson, CT Li, A Class of Discrete Multiresolution Random Fields and its Application toImage Segmentation, IEEE Trans. Pattern Analysis and Machine Intelligence,25(l):42-56,2003.This approach employs Markov random fields (MRF) defined in a multi-resolution structure topartition the target image into homogeneous regions with distinctive features. By exploiting thelocal characteristics, known as the Markovianity of the MRIs, the technique allows the locallabeling and partitioning process to be carried out in a Grid-enabled environment.Dr Chang-Tsun Li is analysing this segmentation technique in the context of Grid-applications,with the aim of being able to identif and employ performance analyses to applications that sharethese core characteristics.Integration with standard middleware toolsWe have begun this at the intra-domain level by modifying Titan (our predictive intra-domainscheduler) so that it now acts as a co-scheduling driver to Condor, see figure below.The advantage of this is that it allows the predictive data derived from PACE to drive the jobsubmission provided by Condor (or indeed any other commodity scheduler). In the case ofCondor this is done by auto-generating ClassAds that force Condor to run a task on a designatedmachine (or set of machines) using the name attribute."J,$iliH.Figure 1: Integration of Warwick middleware tools with standardGrid middleware - phase l: The Titan predictive scheduler hasbeen modified so that it acts as a predictive co-scheduler toCondor.We term this predictive co-scheduling, and it allows traditional Globus/Condor configurations toutilise performance data at the intra-domain level should it exist; if there is no predictive data thenthe job passes to Condor in the same way as before.An additional advantage of the predictive co-scheduling technique described above is the abilityto include quality of service criteria into Condor-based intra-domain scheduling. As it standsCondor has no means of reordering tasks or assigning deadlines so as to improve (or indeedguarantee) the service offered to its users. We have currently implemented three metrics into theTitan co-scheduler, these are makespan (the time to completion), the average advance time (therelationship between a task's execution time and its deadline) and resource utilisation (the abilityto capture the utilisation ofa contributing resource).By providing these metrics at the co-scheduler level, it is possible to monitor execution andexplore execution strategies so that the overall runtime is minimised, the deadlines of tasks aremaintained or so that the resources are utilized to best effect.We are hoping to extend this technique to a multi-domain level using Globus informationservices.BibliographyR Baxter, N Chue, Optimisation Plan for 3D OPT Microscopy Grid, EPCC-IS-3DMG-PLAN 1.0,from EPCC Edinburgh.R Baxter, N Chue, M Parsons, R Baldock, B Hill, J Sharpe, 3D OPT Microscopy Grid - Bringingthe Grid to the Biomedical Workbench, from EPCC Edinburgh.M Brady, eDiamond: Digital Mammography, EPSRC programme GR/S20956/01,2002.D Dillenberger, S Heisig, A Martin, D Gavaghan, hWLM/GRID 2002 project proposal, fromOxford e-Science Centre, 2002.DLG Hill, JV Hajnal, D Rueckert, SM Smith, T Hartkens, K Mcleish, A Dynamic Brain Atlas,available from Derek.Hill@kcl.ac.LrkK Mayes, GD Riley, RW Ford, M Lujan, TL Freeman, The design of a performance steeringsystem for component-based Grid applications, from Centre for Novel Computing, University ofManchester, 2003.GR Nudd, SA Jarvis, Performance-based Middleware for Grid Computing, EPSRC programmeGR/S03058/0t,2002.D Watson, D Gavaghan, M Jirotka, B Collins, A Simpson, M Brady, R Highnam, Critical successfactors and high-levei requirements for eDiamond, November 2002.

A Dynamic Brain Atlas, available from Derek.Hill@kcl.ac.Lrk

Critical success factors and high-levei requirements for eDiamond,

eDiamond:

Performance-based Middleware for Grid Computing, EPSRC programme GR/S03058/0t,2002.

project proposal, from Oxford e-Science Centre,

Review of the e-science demonstrators and their compatibility with the Warwick middleware project

http://wrap.warwick.ac.uk/61311/1/WRAP_cs-rr-394.pdf

Review of the e-science demonstrators and their compatibility with the Warwick middleware project

Abstract

Similar works

Full text

Available Versions

Warwick Research Archives Portal Repository