This paper discusses the use of e-Science Grid in providing
computational resources for modern international High Energy
Physics (HEP) experiments. We investigate the suitability of
the current generation of Grid software to provide the necessary
resources to perform large-scale simulation of the experiment and
analysis of data in the context of multinational collaboration

Khan, A

Wilson, F

English

Brunel University Research Archive

IEEE TRANSACTIONS ON NUCLEAR SCIENCE, VOL. 53, NO. 3, JUNE 2006 1329Grid Applications for the BaBar ExperimentA. Khan, Member, IEEE, and F. WilsonAbstract—This paper discusses the use of e-Science Grid in pro-viding computational resources for modern international High En-ergy Physics (HEP) experiments. We investigate the suitability ofthe current generation of Grid software to provide the necessaryresources to perform large-scale simulation of the experiment andanalysis of data in the context of multinational collaboration.I. INTRODUCTIONTHE BaBar High Energy Physics (HEP) experiment [1] isbased at the Stanford Linear Accelerator Center (SLAC),Stanford, U.S. The experiment investigates the subtle differ-ences between matter and anti-matter by continuously collidingbunches of high-energy electrons and positrons 250 milliontimes per second and searching for the creation of rare B-mesonand anti-B-meson decay. High speed electronics and onlineprocessing rejects unwanted events resulting in a final raw eventrate to tape of approximately 100 Hz with each event requiringkB. The raw data is sent to Padova (Italy), where theevents are reconstructed. The reconstructed events are thenseparated (“skimmed”) into approximately 180 data streamsbased on physics properties using compute farms at Padova,Karlsruhe (Germany) and SLAC (U.S.). These data streamsare made available as datasets for analysis and used by 600researchers based at 74 institutes in 10 countries. Since startingin May 1999, the experiment has recorded 3.5 billion events.In addition to analyzing the data, a major task is the simu-lation of the experiment. At least 3 times as many simulatedevents are needed as data events. Each simulated event takesseconds on a modern processor (2 GHz) and results in 20kB of storage. Jobs of 2000 events are allocated to a distributedsystem based on 25 computer farms located in 5 countries witha total of processors.On each farm a complete BaBar software release needs tobe installed locally or made available to the batch nodes viaAFS [2]. A number of servers must be maintained locally: thelocation of the experimental and simulated data is provided bya MySQL or Oracle database; the experimental and simulatedevents are stored and accessed using the ROOT I/O protocol[3] either directly through Network File System (NFS) or, forheavily used farms, with a load-balanced, fault-tolerant fileserver called Xrootd [4]; an Objectivity [5] object-orientateddatabase is required to store detector conditions, alignment andcalibration constants which are distributed to the jobs via eitheran Advanced Multithreaded Server (AMS) [6] or a NFS. TheManuscript received November 26, 2005; revised February 23, 2006.A. Khan is with the School of Engineering and Design, Brunel University,Uxbridge, Middlesex UB8 3PH, U.K. (e-mail: akram@slac.stanford.edu).F. Wilson is with the Department of Physics, CCLRC, Chilton, Didcot, Ox-fordshire OX11 0QX U.K., (e-mail: fwilson@slac.stanford.edu).Digital Object Identifier 10.1109/TNS.2006.872772system requires a great deal of local customization with eachfarm maintained by a local manager. Load balancing betweenindependent farms has to be done manually. Monitoring of jobsduring production is difficult and there is no way to automati-cally ensure that all resources are available before commencinga production run.This system has run with greater than 98% efficiency andproduced 2.5 billion simulated events in 2004. However, thecoming years will see a tripling in the experimental data rateand a less manpower intensive system must be found to providethree times the resources. A possible approach is based on thetools and middleware provided by Grid systems such as Globus[7] or the LHC Computing Grid Project, LCG [8]. The goal isa production system with a single production manager submit-ting jobs to a worldwide Grid of remote sites with automaticresource allocation, job monitoring, output retrieval and cata-loguing. Grid middleware, working over the Internet, providesthe necessary infrastructure: many Worker Nodes (WN) man-aged by Compute Elements (CE); jobs directed by ResourceBrokers (RB); metadata management by the Replica LocationService (RLS) and high volume of data kept on Storage Ele-ments (SE). We have investigated two possible solutions to thistask. In the first, the minimum amount of Grid software was in-stalled on legacy BaBar farms with data and databases accessedlocally.Only the controlling job description was sent to the grid tech-nology technology using the Globus system. In the second ap-proach, no assumption was made about local resources. The re-quired BaBar software was pre-packaged and installed though aGrid account; data and databases were accessed remotely overthe Wide Area Network (WAN) using the AMS and Xrootd pro-tocols. The first approach offered the benefits of only minimalchanges to pre-existing resources; the second offered the possi-bility of much reduced local customization and support load.II. BABAR SIMULATION PRODUCTION ON THE GRIDTECHNOLOGY TECHNOLOGYIn the UK, the Globus 2.4 gateway was installed on BaBarfarms located at the Universities of Royal Holloway, Bristol andthe Rutherford Appleton Laboratory (RAL). RAL acted as thecontrolling site. The Ganglia [9] monitoring program was in-stalled locally to provide graphical status of the batch workers.Each identical farm comprised of 40 dual CPU Linux machines,a Solaris machine used as an Objectivity server, two interactiveLinux machines, one of which also acted as the OpenPBS [10]server and some NFS mounted storage. The standard produc-tion scripts were modified to submit remotely to the gateway;the jobs were passed onto the batch system and then retrieved atthe end. Initially the Globus file staging functions were used totransport these files to and from the remote site, but this provedunreliable and it was replaced with GridFTP [11].0018-9499/$20.00 © 2006 IEEE1330 IEEE TRANSACTIONS ON NUCLEAR SCIENCE, VOL. 53, NO. 3, JUNE 2006Fig. 1. Resources available for BaBar simulation production on the Italian Grid.The production was automated by creating two daemons atthe central site. The gateway was unable to deal with a largenumber of jobs so a control script was created to monitor thenumber of jobs running and queued at each remote site andsubmit more if the number fell below a certain threshold. It alsointerfaced to the standard BaBar tools to rebuild failed jobs andchoose the next jobs for submission. The second daemon tookthe finished jobs, recovered the data, validated the output andupdated the catalogue.During 2004, this system was deployed at four sites acrossthe UK. 31 million events were produced in nine months on thissystem with peak production rates in excess of 2.5 million eventsper week and more than 100 concurrent jobs at a given remotesite. The failure rate in stable running was below five percent.This system was limited to running on existing BaBar re-sources as the full BaBar software and servers were required.The sites were still separate farms and load balancing was notpossible. No reduction in manpower was achieved as a completelocal installation of the BaBar environment was required withconstant updates to the local databases. A remote control pro-gramme was needed to limit the number of jobs submitted as thestandard Globus gatekeeper spawned processes for every run-ning and queued job which overwhelmed the gatekeeper and re-sulted in an unresponsive system. Job submission failures werealso traced to poor interaction between certain firewalls and theTCP timeout behaviour.The second approach used the full LCG infrastructure andprototyping was performed on the INFNGrid [12] farms directlyinvolved in simulation production for BaBar (Fig. 1). LCG of-fers a number of possible advantages as the middleware was de-veloped with HEP application use cases in mind. It providesan integrated data management system and a Resource Broker(RB) that can direct jobs to compatible resources without in-volvement of the submitter.The LCG model also adds a number of challenges. The BaBarsoftware is not available at the remote site. Therefore the com-plete BaBar software was reduced to a core subset that was pack-aged as a tar archive of about 37 MB (130 MB uncompressed).The package was then distributed on each Grid Worker Node byusing procedures based on the LCG middleware to install anduninstall new software and to publish new tags. A site manager,mapped as a special user, was able to use these procedures bysimply submitting specific jobs on the grid technology choosingwhich sites must be involved.Three sites (Ferrara, Padova, Naples) were configured to pro-vide Objectivity and Xrootd services that could be accessedfrom the grid technology Worker Nodes over the WAN usingAMS and the Xrootd protocol. The typical amount of conditionsdata read by a single simulation job was about 2% of the totaland the use of the WAN reduced the CPU efficiency by about5%, which was acceptable.Standard BaBar software tools for simulation productionwere installed on the LCG User Interface (UI) and properlyconfigured. These tools were needed to merge several collec-tions of homogeneous events into a single collection, transferevents to SLAC, and to update the bookkeeping database atSLAC.A Job Description Language (JDL) [13] file which is em-bedded in a script sets up all data needed and specifies theparameters for the simulation including type of events, runnumber, background triggers to use, number of events anddetector conditions. Jobs were then submitted through a UIto a Resource Broker specifically installed for BaBar, locatedin Ferrara. The RB was able to manage Italian and Europeanresources directly involved in BaBar. The RB performedmatchmaking between resources available on each site of thegrid technology, published by Compute Element (CE), andjobs requirements (memory, specific software release andtype of the batch queue). Jobs were sent to CEs satisfying therequirements, and distributed to WNs, where the executionstarted. The simulated events were saved as ROOT I/O filesand transferred from WNs to a Storage Element (SE) locatedin Ferrara. A conceptual schema of the flow described above isshown in Fig. 2.During initial tests a single condition database installed inFerrara was used as AMS server while jobs were submitted onFerrara, Napoli, Trieste, Catania, Bari and Padova. A modifiedversion of AMS, written at SLAC [14], was designed to handlemore connections. It was installed in Ferrara and its perfor-mance was compared to the standard AMS. Both AMS serversused 32 threads to handle client requests. Stress tests, performedby submitting from 100 to 250 jobs of 2000 events each, showedKHAN AND WILSON: GRID APPLICATIONS FOR THE BABAR EXPERIMENT 1331Fig. 2. BaBar simulation production model implemented in Italy.TABLE IDATA READ FROM AMS LOCATED IN FERRARAthat standard AMS supported about 30 concurrent simulationjobs before failing to respond. The number of failed submis-sions increased with increasing load, reaching almost 50% when80 clients contacted the server simultaneously. Performance im-proved using the SLAC AMS server. The server was able tohandle about 75 clients before jobs failure occurred.The maximum number of AMS connections a client can gen-erate could be configured in the job script. Limiting the numberof connections increased the execution time for some jobs whileallowing more jobs to contact the AMS in parallel. A good bal-ance between efficiency and stability was achieved, setting themaximum number of connections to 20 for each client.After these initial tests, 2 tests were performed accessing theAMS server at different sites. For every test, 200 jobs were sub-mitted, each with 2000 events in order to simulate a real produc-tion flow. For both tests, the time spent to complete jobs on eachsite was measured. The maximum number of concurrent clientsconnected during production was taken into account. The linkspeeds to Bari, Ferrara, Napoli and Padova were 15, 16, 32 and160 Mb/s respectively.Data were read using the AMS server installed in Ferrara,while production jobs have been submitted on Worker Nodeslocated in Ferrara, Padova, and Bari. Most of the jobs run onPadova due to the high number of WNs provided. About 99%jobs successful completed (Table I), a success rate comparableto the non-GRID production.In the second test, data were read remotely, using the AMSserver installed in Naples from production jobs submitted onWorker Nodes located in Ferrara, Padova and Bari (Table II).TABLE IIDATA READ FROM AMS LOCATED IN NAPLESDue to the fast network link, jobs at Padova completed morequickly than jobs submitted on other sites. The quality of net-work access and data location played a basic role in turn aroundtimes and is an important parameter in choosing the optimal run-ning conditions. The whole system was tested producing realallocations data for Monte-Carlo (more than 3,000,000 events),successfully completing the whole production cycle for about98% of the total jobs.III. ANALYSIS OF THE GRID TECHNOLOGYThe simulation of data is a much more tractable problem thanthe analysis of data. Data analysis requires 100% reliability ifthe results are to have any meaning and needs to access hugeamount of data, which has to be located and accessed remotely.The output differs in form and size depending on the type ofanalysis. To catalogue and make this non-standardized outputavailable needs a flexible system. To test an analysis environ-ment, a prototype farm based on LCG middleware softwarewas created to test simple stand-alone analysis programs thatcould handle data location and job management. Initial testingwith specialized Grid-analysis infrastructures using AliEn [15]showed possibilities, but also the need for greater extensibilityand stability. We therefore created a specific BaBar job submis-sion script called EasyGrid.IV. EASYGRID JOB SUBMISSION SYSTEM FOR ANALYSISThe EasyGrid Job Submission software [16] is an interme-diate layer between the grid technology, where the resourcescan be found, and the user, who has a pre-developed software1332 IEEE TRANSACTIONS ON NUCLEAR SCIENCE, VOL. 53, NO. 3, JUNE 2006analysis (written in C++), which she runs on a chosen dataset.EasyGrid’s first task is to find what event files are in the dataset.For each dataset there is a metadata file containing the names ofthe event files. These physical files are registered with the RLS,with several logical file names in assigned to them as aliases,identifying which CEs contain copies of that dataset. If a CEholds any of the files in a dataset it holds all of them. Searchingall the aliases for a dataset name provides a list of CEs to whichjobs can be submitted.The next stage is generation of all necessary information tosubmit the jobs on the grid technology. This is done by the GEn-erator of Resources Available (Gera) which produces the JDLfiles, the script with all necessary tasks to run the analysis re-motely at a WN, and some Grid dependent analysis parameters.The JDL files define the input sandbox with all necessary filesto be transferred, and a WN load-balance algorithm allocatesthe task to machines that provide the most effective resources.When the task is delivered in the WN, scripts start running toinitialize the specific BaBar environment, and the analysis soft-ware is downloaded. The analysis executable is allocated in theSE and its logical file name (LFN) is also catalogued in the RLSso any WN need download it only once.Users can follow up the process by querying job status. If thejob is done, a task recovering results in the user’s directory isperformed automatically. If the job was aborted in the process,the diagnostic listing is stored in the history file for further anal-ysis.V. CONCLUSIONThe simulation of HEP events is now feasible using Grid tech-nology. Reliability is beginning to approach that of local self-contained compute farms. The grid technology can be interfacedto legacy software and procedures. As most non-trivial HEP pro-grams require access to centralized or distributed resources, ourprototyping shows that the major architectural problems con-cern the installation of these servers and their efficient accessover the WAN. EasyGrid opens the possibility to use the gridtechnology for analysis using the data accumulated by BaBarover five years. Future work will concentrate on simplifying theidentification of data, providing a common environment on allGrid resources and providing servers to provide regional accessover the WAN to centralized databases.ACKNOWLEDGMENTThis work builds on a large body of development by theBaBar Computing Group, and would not have been possiblewithout a strong collaborative effort.REFERENCES[1] B. Aubert, “The BaBar detector,” Nucl. Instrum. Methods, vol. A 479,pp. 1–116, 2002.[2] J. H. Howard, “An Overview of the Andrew file system,” in USENIXConf. Proc., Winter 1988, Carnegie Mellon University.[3] R. Brun and F. Rademakers, “ROOT—An object oriented data analysisframework,” (in See also) Nucl. Instruments & Methods, vol. A 389, pp.81–86, 1997 [Online]. Available: http://root.cern.ch/[4] A. Hanushevsky, “The next generation root file server,” in Proc. CHEP,Interlaken, Switzerland, Sep. 2004.[5] Objectivity Technical Overiew (in Objectivity Inc.) , 1996.[6] J. Becla, Setting up AMS 2002, AMS [Online]. Available: http://www.slac.stanford.edu/BFROOT/www/Public/Computing/Databases/ex-perts/setupAMS.shtml[7] Int. J. Supercomputer Applications (in The Globus alliance) vol. 15,no. 3, 2001.[8] “LHC computing grid project,” in Proceedings of CHEP04 (in CERN),Interlaken, Switzerland, Sep. 2004.[9] M. L. Massie, “The ganglia distributed monitoring system: Design, im-plementation and experience,” Parallel Computing, vol. 30, no. 7, July2004.[10] OpenPBS [Online]. Available: http://www-unix.mcs.anl.gov/openpbs/[11] W. Allcock, “GridFTP: Protocol extensions to FTP for the Grid,”Global Grid Forum GFD-R-P.020.[12] INFN Production Grid (in INFNGrid: GridIt) [Online]. Available:http://www.grid.it/[13] Job Description Language: DataGrid-01-TEN-0102-0_2 (in Datamat)2001.[14] A. Hanushevsky, Running Multiple AMS Servers per Host (in SLACAMS) [Online]. Available: http://www.slac.stanford.edu/~abh/Objec-tivity/Misc/mult_ams.html 2002[15] F. Radenmakers, “Global distributed parallel analysis using proof andalien,” in Proceedings of CHEP04, Interlaken, Switzerland, Sep. 2004.[16] J. C. Werner, HEP Analysis, Grid and Job Submission [Online]. Avail-able: http://www.hep.man.ac.uk/u/jamwer/

An Overview of the Andrew ﬁle system,”

Global distributed parallel analysis using proof and alien,”

GridFTP: Protocol extensions to FTP for the Grid,” Global Grid Forum GFD-R-P.020.

HEP Analysis, Grid and Job Submission [Online].

INFN Production Grid (in INFNGrid: GridIt) [Online].

Language: DataGrid-01-TEN-0102-0_2 (in Datamat)

LHC computing grid project,”

LHCcomputinggrid project,”inProceedings ofCHEP04(inCERN),

M.L.Massie,“Thegangliadistributedmonitoringsystem:Design,implementation and experience,”

ROOT—An object oriented data analysis framework,” (in See also)

Running Multiple AMS Servers per Host

Setting up AMS

Supercomputer Applications (in The Globus alliance)

The BaBar detector,”

The ganglia distributed monitoring system: Design, implementation and experience,”

The next generation root file server,” in

Grid applications for the BaBar experiment

http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.410.1207

Grid applications for the BaBar experiment

Abstract

Similar works

Full text

Available Versions

Brunel University Research Archive