The total dataset produced by the BaBar experiment at the Stanford Linear Accelerator Center (SLAC) currently comprises roughly$3times 10^9$data events and an equal amount of simulated events, corresponding to 23 Tbytes of real data and 51 Tbytes simulated events. Since individual analyses typically select a very small fraction of all events, it would be extremely inefficient if each analysis had to process the full dataset. A first, centrally managed analysis step is therefore a common pre-selection (‘skimming’) of all data according to very loose, inclusive criteria to facilitate data access for later analysis. Usually, there are common selection criteria for several analysis. However, they may change over time, e.g., when new analyses are developed. Currently,$cal 

Fritsch, M

Gradl, W

Khan, A

Mommsen, RK

Petzold, A

Roethel, W

Smith, DA

English

A. Khan

R.K. Mommsen

W. Gradl

M. Fritsch

A. Petzold

W. Roethel

D.A. Smith

Crossref

Performance of the Distributed Central Analysis in BaBar

Brunel University Research Archive

2876 IEEE TRANSACTIONS ON NUCLEAR SCIENCE, VOL. 53, NO. 5, OCTOBER 2006Performance of the DistributedCentral Analysis in BaBarA. Khan, Member, IEEE, R. K. Mommsen, W. Gradl, M. Fritsch, A. Petzold, W. Roethel, and D. A. SmithAbstract—The total dataset produced by the BaBar experimentat the Stanford Linear Accelerator Center (SLAC) currently com-prises roughly 3 109 data events and an equal amount of simu-lated events, corresponding to 23 Tbytes of real data and 51 Tbytessimulated events. Since individual analyses typically select a verysmall fraction of all events, it would be extremely inefficient if eachanalysis had to process the full dataset. A first, centrally managedanalysis step is therefore a common pre-selection (‘skimming’) ofall data according to very loose, inclusive criteria to facilitate dataaccess for later analysis. Usually, there are common selection cri-teria for several analysis. However, they may change over time,e.g., when new analyses are developed. Currently, (100) suchpre-selection streams (‘skims’) are defined. In order to providetimely access to newly created or modified skims, it is necessaryto process the complete dataset several times a year. Additionally,newly taken or simulated data has to be skimmed as it becomesavailable. The system currently deployed for skim production isusing 1800 CPUs distributed over three production sites. It waspossible to process the complete dataset within about 3.5 months.We report on the stability and the performance of the system.Index Terms—Data handling, data management, data pro-cessing, distributed computing.I. INTRODUCTIONTHE BaBar detector [1] at the Stanford Linear AcceleratorCenter (SLAC) in the US collects data from colli-sions at the B meson resonance. 230 fb of physics qualitydata were collected in the years 1999 to 2004. This correspondsto events or 23 Tbytes of reconstructed quantities, notcounting raw data ( Tbytes) or multiple reconstructionpasses. In addition, Monte Carlo (MC) events havebeen simulated and reconstructed, adding another 51 Tbytes ofreconstructed data.Manuscript received January 3, 2006; revised June 16, 2006.A. Khan is with the School of Engineering and Design, Brunel University,Uxbridge, Middlesex UB8 3PH, U.K (e-mail: akram@slac.stanford.edu).R. K. Mommsen was with the University of California, Irvine, Irvine,CA 92664 USA. He is now with the University of Manchester, ManchesterM13 9PL, U.K. and Fermilab, Batavia, IL 60510 USA (e-mail: mommsen@slac.stanford.edu).W. Gradl is with the Department of Physics, University of Edinburgh,Edinburgh EH9 3JZ, U.K. (e-mail: wgradl@slac.stanford.edu).M. Fritsch is with the Inst. f. Experimentalphysik 1, Universitätsstrasse 150,Ruhr Universität Bochum, D-44780 Bochum, Germany (e-mail: miriam@ep1.ruhr-uni-bochum.de).A. Petzold is with the Inst. f. Kern- u. Teilchenphysik, Technische Universität,Dresden, D-01062 Dresden, Germany (e-mail: a.petzold@physik.tu-dresden.de).W. Roethel is with the University of California, Irvine, Irvine, CA 92697USA (e-mail: roethel@slac.stanford.edu).D. A. Smith is with the Stanford Linear Accelerator Center, Menlo Park, CA94025 USA (e-mail: douglas@slac.stanford.edu).Digital Object Identifier 10.1109/TNS.2006.881737A typical physics analysis selects only a tiny fraction of allevents. Therefore, it is extremely inefficient for each analysis toloop over all data. Additionally, allowing many analysis jobs toaccess the central event store puts a high load on the system andrequires a large local computing farm to provide the necessaryCPU power and disk capacity. In order to circumvent these is-sues, the concept of a centrally managed pre-selection of eventswas introduced as first step in the analysis chain (see Fig. 1).The central analysis separates the events into streams,called ‘skims’, according to loose, inclusive physics criteria de-fined by the analysis groups or individual analysts. Therefore,the analysis-specific skim contains a higher purity of interestingevents for a given analysis. On average, every event is found in2 skims. The reconstruction quantities for selected events can beduplicated and are included in the skim. Thus, they can be easilydistributed to different computing sites, grouped according tophysics interests. At the same time, analysis-specific quantitiescan be added to the skims. This empowers the analysts to per-form the final data fitting without a massive ntuple productionon their own.The skim definitions change over time as a better under-standing of the physics and the detector evolves. Also newskims are devised, as new analysis topics emerge. Therefore,it is necessary to have several skim production cycles peryear. Additionally, newly taken or simulated data needs to beprocessed as it becomes available. Since all analyses depend onthe central skim production, a timely and reliable processing isessential.II. DATA ORGANIZATIONIn fall 2003 a new computing model [2] was deployed inBaBar. All data except the raw data is stored as ROOT [3] files.The ROOT files are administered by a relational database. Thebasic entity is a collection, uniquely defining a set of ROOT fileswith a common denominator. A dataset holds multiple collec-tions with similar content. A dataset is dynamically defined byproperties associated with a collection. E.g., a dataset can holdall collections from a given run period. Datasets are updated au-tomatically when new collections fitting the corresponding def-inition become available. Tags, similar to CVS tags, are usedto freeze a dataset. Tags can be used to mark a common datasample, e.g., the data used for analyses targeted at a specificconference.The ROOT files are accessed using the xrootd protocol [4].This protocol is implemented in the ROOT framework and al-lows a distributed access to the data. Additional tools have beendeveloped to copy local files into the xrootd event store. Forthe central analysis, two types of xrootd event stores are used: a0018-9499/$20.00 © 2006 IEEEKHAN et al.: PERFORMANCE OF THE DISTRIBUTED CENTRAL ANALYSIS IN BaBar 2877Fig. 1. Sketch of the analysis flow in BaBar. The reconstructed data originating from the BaBar detector and Monte Carlo (MC) simulation is stored in a centralevent store. It is inefficient and requires large local resources to allow all analysis applications to directly access the central event store. Therefore, a central analysisas first step in the analysis processes splits the physics events in O(100) skims according to physics criteria defined by the analysis groups. These skims can bedistributed to several remote farms where each analysis can access a purified sample of interesting events.public event store where the reconstructed events and the skimsare stored, and for temporary storage of the intermediate outputproduced during the skim production.III. CENTRAL ANALYSISThe central analysis consists of several tasks handling dif-ferent types of data. Each task uses as input one or severaldatasets. The tasks are distributed to different computing cen-ters and assigned priorities for the processing. Each task pro-cesses all collections belonging to its associated datasets. As theprocessing of one collection typically takes several CPU days,it turned out to be more efficient to split each collection intosmaller chunks that can be processed within 4 to 5 hours. Thisallows a more flexible handing of the CPU queues. It also savesa considerable amount of CPU time if a job crashes due to com-puting problems. Each job produces all skims which are definedfor the given skim cycle. Each skim corresponds to a ROOTfile, that is typically less than a few kbytes, since only a tinyfraction of the events is selected. Mass storage systems ineffi-ciently handle small files and there is a considerable overheadin opening many small files during the analysis. Therefore, asecond step merges several files belonging to the same skim andhaving some commonality into larger files. These files are im-ported into the mass storage system and their properties are in-serted into the bookkeeping database, where the correspondingFig. 2. The Task Management controls the job creation and batch execution aswell as the bookkeeping of processed input collections and produced temporaryand final collections.collections are created and added to datasets. The analyst canaccess the skimmed collections using datasets.The central analysis needs a huge amount of CPU powerto process all available data in a reasonable time. In order toachieve a timely production, the load is distributed over multiple2878 IEEE TRANSACTIONS ON NUCLEAR SCIENCE, VOL. 53, NO. 5, OCTOBER 2006TABLE IPROCESSED EVENTS AND OUTPUT PRODUCED DURING THE FIRST FIVE SKIM CYCLEScomputing centers. Up to now we used the computing centersat the University of Padova/INFN (Italy), GridKa (Karlsruhe,Germany) and SLAC (California, US). The assignment of tasksis agreed upon before a given skimming cycle starts. The datadistribution is done using multi-stream file transfer tools. Inorder to facilitate the bookkeeping of processed and availablecollections, all input data was transfered from SLAC to the re-mote sites and the finished skim collections were copied backinto the SLAC event store before redistributing them to the anal-ysis centers.The Task Management (TM) framework [5] is used tomanage the central analysis (Fig. 2). The TM handles thejob submission and control. It creates the skimming jobs forthe input collections, assuring that each input collection isonly processed once. It submits them to the batch system andchecks for their successful execution. In case of job failures, abasic failure classification is done and jobs likely to succeedon a further try are resubmitted. The TM keeps track of thetemporary output collections and determines which ones canbe merged together. It also handles the creation and submissionof the merge jobs as well as the final import into the event storeand bookkeeping database.Further tools were developed during the skim productionto improve the flexibility in automatically handling multipleparallel tasks on the batch system and to keep the batch queuesfilled. Monitoring tools were used to create websites withprogress and status information. Additional tools were neededfor managing the data distribution to the remote sites and fordatabase consistency checks.IV. PERFORMANCEGiven that all analyses depend on the availability of skims,the timely production of skims is essential for the success of thephysics program. The aim is to have 3 to 4 skimming cycles peryear. In each cycle, only modified or newly defined skims needto be produced unless a new reconstruction was performed. Fornewly available data all defined skims need to be produced.Table I lists the skim cycles completed so far. The first skimcycle R14 was started at the beginning of 2004 and producedall skims defined at that time. The R14a skim cycle fixed a fewbugs present at the beginning of data skimming. The subsequentR16a skim cycle started in late December 2004 producing 91newly defined and 70 redefined skims. The following two skimcycles produced a total of 38 new skims. Currently, a full recon-struction and new simulation effort is ongoing which is beingskimmed as the reconstructed data becomes available.Fig. 3. Event rate for data skimming as function of the number of eventsprocessed by one job, comparing the different CPU types available at SLAC.The default number of events processed by one job is 20000. The event rate isusing wall clock time and includes the overhead for initializing and checkingof the output.In the following, we will look at the performance and relia-bility of the last large scale skim production cycle R16a.The batch queue used at SLAC consists of approximately450 PIII 1.4 GHz, 300 Xeon 2.6 GHz, and 220 Opteron1.8 GHz machines. The actual number of CPUs varied due tomachine failures and queue allocation policies. The Opteronmachines became available during January 2005 and theirnumber gradually increased over the following month. Allmachines are dual-CPU machines with 2 Gbytes memory.Fig. 3 shows the processing rate versus the number of eventsprocessed by a skim job running over real data. The processingtime is wall clock, i.e., it includes all overheads from startingjob and checking the output for consistency, but does notincluding the batch queuing time. It can be clearly seen thatbelow 5000 events the overhead for initializing and checkingdominates, while for larger chunks the processing rate is con-stant. The skim performance of PIII and Xeon processors iscomparable, while the rate of the Opteron machines is 1.8 timeshigher. A similar behavior is observed for MC events. Theprocessing rate depends on the type of simulated events beingprocessed. The processing rate is reduced by a factor 5 forthe most demanding MC (‘B generics’) which include the fullspectrum of possible B decays overlaid with background data.Fig. 4(a) shows the number of successfully skimmed eventsversus time. After a slow startup phase over Christmas 2004, thereal data was skimmed within 1.5 months. The bulk of the MCskimming was finished two months later. Roughly 10% of theKHAN et al.: PERFORMANCE OF THE DISTRIBUTED CENTRAL ANALYSIS IN BaBar 2879Fig. 4. Performance of the last large scale skim production cycle R16a. On the left we show the cumulative number of events skimmed versus time (shown inmonth/day format for the years 2004/2005). After a slow startup phase over Christmas 2004, the real data was skimmed within 1.5 months, while the bulk ofsimulated Monte Carlo events was skimmed 2 months later. On the right the number of CPUs used at SLAC versus time during the main production period in 2005is shown. The number of available CPUs increased during this time and except for short periods of time, all CPUs were busy. (a) Number of successfully skimmedevents versus time. (b) Number of CPUs used at SLAC versus time.Fig. 5. Scalability of the system shown for data (left) and Monte Carlo (right). The number of processed events per day scales nicely with the number of CPUsused. The linear fit to the data points yields 10M (2M) events/day/100 CPUs for data (Monte Carlo).jobs had to be resubmitted due to machine failures or I/O prob-lems. Fig. 4(b) depicts the queue usage at SLAC during the mainproduction period. It shows that it was possible to constantly useall available CPUs at SLAC. The occasional lower CPU usagewas due to the limitations in the available disk space used forstoring the temporary collections.V. SCALABILITYThe number of CPUs devoted to a given task changed overtime. This allows the investigation of the scalability of thesystem as function of used CPUs. Fig. 5 shows the numberof events being successfully processed per day versus thenumber of CPUs utilized. We observe a linear behavior. Thescatter of the points around the fitted line is mainly due tothe fact that we did not correct for the different processingspeeds of the machines involved. The linear fit yields 10 M(2 M) events/day/100 CPUs for data (Monte Carlo). This rateincludes the time needed for initializing the jobs, checking theproduced output, and rerunning of failed jobs. It also includesthe overheads from bookkeeping, but does not account forbatch queuing time.VI. CONCLUSIONThe concept of a central analysis within the new computingmodel based on ROOT files has been successfully implemented.Since its deployment in fall 2003, 5 skim production cycles havebeen completed. We demonstrated that it is possible to processthe huge data sample in a reasonable time and to provide skimsto the analysts. With a special effort before a major conference,we succeeded to have skims available for data taken only eightdays before. The system enables a reliable and large scale oper-ation and shows an excellent scaling behavior. Given the recentexperience, the target of running 3 to 4 skim cycles per year ina timely fashion is achievable. We successfully processed morethan events and produced 220 Tbytes of skims usingin excess of 600 000 CPU-days at three computing centers.2880 IEEE TRANSACTIONS ON NUCLEAR SCIENCE, VOL. 53, NO. 5, OCTOBER 2006For future development, skim production in the grid is an in-teresting option. We are already running at three different com-puting centers, and the Task Management system demonstratedthe feasibility for running in a distributed environment. The cen-trally defined skim applications are more stable than individualuser analyses which makes the code distribution easier. In con-trast to simulation tasks which already run routinely in the grid,skimming could be an application to test the data access withinthe grid, without the difficulties of unpredictable access patternsimposed by normal analysis tasks.ACKNOWLEDGMENTThis work builds on a large body of development by theBaBar Computing Group, and would not have been possiblewithout a strong collaborative effort.REFERENCES[1] B. Aubert, “The BaBar detector,” Nucl. Instrum. Methods Phys. Res.A, vol. A479, pp. 1–116, 2002.[2] D. N. Brown, “The new BaBar analysis model,” in Proc. CHEP ’04,Interlaken, Switzerland, Sep. 2004.[3] R. Brun and F. Rademakers, “ROOT—An object oriented data anal-ysis framework,” Nucl. Instrum. Methods Phys. Res. A, vol. A389, pp.81–86, 1997.[4] A. Dorigo, P. Elmer, F. Furano, and A. Hanushevsky, “XROOTD—Ahighly scalable architecture for data access,” in Proc. WSEAS Conf.,Prague, Czech Republic, Mar. 2005.[5] W. Roethel, “The BaBar analysis task manager,” in Proc. CHEP ’04,Interlaken, Switzerland, Sep. 2004, pp. 348–353.

Performance of the distributed central analysis in BaBar

Abstract

Similar works

Full text

Available Versions

Crossref

Brunel University Research Archive