This document summarises the status of the existing grid infrastructure and functionality for the high-energy physics experiment CMS and the expertise in operation attained during the so-called ”Computing, Software and Analysis Challenge” performed in 2006 (CSA06). This report is especially focused on the role of the participating computing centres in Germany located at Karlsruhe, Hamburg and Aachen

Ernst, M.

Flossdorf, A.

Hof, C.

Kress, T.

Quast, G.

Rabbertz, K.

Scheurer, A.

English

MPG.PuRe

2007German e-ScienceAvailable online at http://www.ges2007.deThis document is under the terms of theCC-BY-NC-ND Creative Commons AttributionChallenges of the LHC Computing Grid by theCMS experimentA. Scheurer1, M. Ernst3, A. Flossdorf3 C. Hof2, T. Kress2, K. Rabbertz1, G. Quast11 Institut für Experimentelle Kernphysik, Universität Karlsruhe,Wolfgang-Gaede-Str. 1, 76131, Karlsruhe, Germany2 III. Physikalisches Institut, RWTH Aachen, 52056 Aachen, Germany3 DESY, Notkestrasse 85, 22607 Hamburg, GermanyAbstractThis document summarises the status of the existing grid infrastructureand functionality for the high-energy physics experiment CMS and theexpertise in operation attained during the so-called ”Computing, Soft-ware and Analysis Challenge” performed in 2006 (CSA06). This reportis especially focused on the role of the participating computing centresin Germany located at Karlsruhe, Hamburg and Aachen.1 IntroductionIn preparation for the enormous amounts of data expected from the futurelarge hadron collider (LHC) presently under construction at the European lab-oratory for particle physics, CERN, a grid infrastructure is being set up by thephysics communities in the participating countries. The world-wide LHC Com-puting Grid (WLCG) is based on tier-structured layers of one Tier0 at CERNand several Tier1, Tier2 and Tier3 centres distributed all around the globe. TheGerman part of the CMS grid infrastructure relies heavily on the two Helmholtzcentres Forschungszentrum Karlsruhe as Tier1 and DESY Hamburg as well ason installations at the universities Karlsruhe and Aachen. The emphasis of thisreport lies on so-called service challenges performed in 2006 where the full func-tionality of the whole chain from data processing at the Tier0, data distributionto the Tier1 and Tier2 centres up to organised data re-processing and morechaotic physics analyses by single users was exercised.2 The CMS Computing ModelThe CMS computing model [1] relies on the hardware infrastructure and theservices offered by the world-wide LHC Computing Grid (WLCG) [2, 3]. Itis based on four hierarchically-ordered layers of one Tier0 situated at CERN,seven Tier1 centres in Europe, Northern America and Asia, about 30 Tier2 andnumerous Tier3 centres distributed equally around the world, see figure 1. Thedata flow from the source, the CMS detector, to the desktops of the analysingphysicists has to pass many different stages:2 A.Scheurer,M.Ernst,A.Flossdorf,C.Hof,T.Kress,K.Rabbertz,G.QuastKarlsruheAachenFermilab /TaiwanRutherford Lab /United KingdomSpanienGridKa /GermanyHamburgTier 0Tier 1Tier 2/3USAIN2P3 /FrankreichASGC /PIC /ItalienCNAF /T2sT3sT2sT3sT2sT3sT2sT3sT2sT3sT2sT3sT2sT3sT2sT3sT2sT3sT2sT3sT2sT3sT2sT3sFigure 1: Overview of the WLCG/CMS tier structure, including Tier0, all sevenTier1 and several Tier2 and Tier3 sites.• The Tier0 at CERN collects the data accepted by the trigger system ofthe experiment and performs a first reconstruction pass at a rate of 40Hz using the new CMS software framework described in section 3.4. Theprocessed data is stored permanently on magnetic tapes and temporarilyon disk-pools for further distribution.• These data are distributed to the regional Tier1 centres, where a second,distributed copy is archived on magnetic tape for future reprocessing. TheTier1 also receives a full copy of the particle information as reconstructedin the detector, on which most physics analyses will be performed.• Calibration jobs are executed at the Tier0, Tier1 and Tier2 centres.• Re-reconstruction of existing datasets and skim jobs (generation of smallerdata samples matching certain selection criteria) are performed at theTier1s.• Selected data are exported to the Tier2 centres, where analysis jobs byindividual physicists are executed.• The Tier3 centres are not explicitly defined in the computing models. Theyprovide local resources for the analysing physicists, in particular for inter-active analysis, software development, testing and grid access and the finalinterpretation of the analysis results.3 The Computing, Software and Analysis Challenge 2006In order to test the readiness of CMS for the upcoming startup of LHC, aso-called Computing, Software and Analysis Challenge (CSA06) was performedin autumn 2006. More information than given below can be found in the CSA06Summary Report [4].GES 2007 Challenges of the LHC Comp. Grid by the CMS experiment 33.1 Motivation and GoalsSince the buildup processes of the CMS computing resources are still ongoingand far from being finished, CSA06 was designed to operate at a level of 25% ofthe resources foreseen for 2008 during four weeks of operation. Large datasetshad to be simulated as input for the Tier0 reconstruction. The success of CSA06was measured by pre-defined metrics (the parenthesised numbers indicate thethreshold for being considered a success):• Automatic data transfer from Tier0 to Tier1 and Tier1 to Tier2 via theCMS data placement tool PhEDEx [5], as well as automatic transfer ofskim job output data to the Tier2 centres.• Read and write access to the offline database at CERN (Oracle) via thedatabase caching service Frontier/Squid [6, 7], and running of re-reconstructionjobs generating datasets based on updated database information.• Submission of user jobs via the grid job submission tool CRAB [8] andusing the CMS-developed Dataset Bookkeeping Service (DBS) and theData Location Service (DLS).• Number of participating Tier1 sites: 7 (5) with an uptime > 90%.• Number of participating Tier2 sites: 20 (15).• Weeks of running at the sustained rates: 4 (2).• Tier0 efficiency during the best 2 weeks: 80% (30%).• Number of running grid jobs at Tier1 and Tier2 centres per day, each jobrunning about 2 hours: 50 000 (30 000).• Grid job efficiency: 90% (70%).• Data serving capability for the jobs at each participating site: 1 MB/secper execution slot (Tier1: 400 MB/sec, Tier2: 100 MB/sec).• Data transfer rates from Tier0 to Tier1 tape system individually vary be-tween 10 - 50 MB/sec (5 - 25 MB/sec).• Data transfer rates to the Tier2 centres: 20 MB/sec (5 MB/sec).The tools mentioned here are described in section3.5.3.2 Participating SitesTier0: Based at CERN, the Tier0 is the core centre within the CMS computingmodel. As mass storage it provided two standard Castor pools, an input poolwith about 64 TB and an export pool with 155 TB of disk space. 700 dual-CPUworker nodes with a system clock of 2.8 GHz and 2 GB of memory formed thebatch system running under LSF (Load Sharing Facility) [9] on SLC4 (ScientificLinux CERN version 4).Tier1: All seven CMS Tier1 centres participated in CSA06 and provided 25%of the resources descibed in the Memorandum of Understanding (MoU) [3]. Theestimated requirements to reach the goals of the service challenge were about300 CPUs for processing and about 70 TB of storage capacity for a nominalTier1.4 A.Scheurer,M.Ernst,A.Flossdorf,C.Hof,T.Kress,K.Rabbertz,G.QuastTier2: In total 27 Tier2 centres participated in CSA06. The estimated require-ments for a nominal Tier2 were about 100 CPUs and 10 TB of disk space. Thevariance between the Tier2 centres, however, is rather wide.3.3 German SitesWith GridKa, the German Tier1 centre, and a federated Tier2 consisting ofinstallations at DESY and RWTH Aachen, the German groups will be well po-sitioned to competitively contribute to the physics analysis. CSA06 used gLiteversion 3.0 as grid middleware, and the CMS-specific services described in sec-tion 3.5 were installed on the operating system Scientific Linux CERN version 3(SLC3) at each site.Tier1 GridKa: GridKa provided about 70 TB of storage space and 250 CPUsduring CSA06. The storage system is based on dCache [10] with four pool nodesdedicated to CMS. The network connection between CERN and GridKa con-sisted of one 10 GBit link via optical fiber. As the CMS computing model alsoforesees data transfers to any Tier2 from Tier1 centres, corresponding links tonumerous Tier2 centres via standard Internet connections have been established.Tier2 DESY and Tier2/3 Aachen: The German Tier2 is a federation of twoindependent grid installations at DESY, Hamburg and the university of Aachen.DESY provided 100 CPUs and 35 TB of dCache storage space. The spacialproximity of the Aachen Tier2 and Tier3 allows to gain from potential synergyeffects due to sharing parts of the same hardware architecture. Aachen providedabout 35 workernodes with 94 cores in mixed dual- and quad-core machines,1 GB of RAM per core. The batch system is running Maui/Torque [11, 12].The storage is handled by a dCache system with a capacity of more than 5 TBdedicated to CSA06.Tier3 Karlsruhe: The Tier3 cluster at Karlsruhe is provided by the Institut fürExperimentelle Kernphysik. The cluster structure is quite heterogeneous withan order of 30 worker nodes with a mixed single and dual CPU setup with about1 or 2 GB of memory per core. They run under the Maui scheduler and theTorque batch system. The storage capacity adds up to 8 TB of nfs shared diskspace for CMS.3.4 The CMS Offline SoftwareThe physics-related software of CMS, i.e. simulation and reconstruction code,has undergone heavy development during preparation of CSA06. The CMSsoftware is managed centrally using grid jobs for installation and validation.Nonetheless, frequent software updates added some instability even during themain phase of CSA06.GES 2007 Challenges of the LHC Comp. Grid by the CMS experiment 53.5 Production and Grid ToolsCRAB - CMS Remote Analysis Builder: CRAB allows access to the re-motely stored data and thus analysing it via the grid infrastructure. With thehelp of the CMS data management services DBS and DLS it determines thelocation of the data, prepares the needed user code for the analysis task andmanages the interaction with the gLite middleware from job submission over jobstatus queries up to job output retrieval.DBS - Dataset Bookkeeping System: At CMS, each dataset is divided upin several data blocks, whereas each block contains on the order of some hun-dred files represented by Logical File Names (LFN). The Dataset BookkeepingSystem is responsible for the proper mapping between files and blocks as well asproviding some processing related information like the size or number of eventsfor each block.DLS - Data Location Services: Based on the Local File Catalog (LFC) in-frastructure provided by the EGEE grid project [13], DLS is responsible for themapping between data blocks and grid sites physically hosting them.TFC - Trivial File Catalog: Since DBS and DLS only provide the locationsof a specified data block in the grid and the LFNs of the contained files, theTFC is responsible for the mapping of each LFN to a physical filename on eachsites storage system. The TFC is a xml-based mapping file hosted on each tiersite.PhEDEx - Physics Experiment Data Export: The CMS data placementtool is responsible for physically transferring the data between the different sites.Although PhEDEx is capable of using the basic grid file transfer protocols likegridFTP, only more advanced protocols like SRM (Storage Resource Manager)and FTS (File Transfer Service) have been used during CSA06.3.6 Data ProcessingThe workflow at the Tier0 during CSA06 covered prompt reconstruction, cre-ation of special data streams for detector alignment and calibration (AlCaReco)and data for analysis (AOD), merging them to larger output files, injection intothe Dataset Bookkeeping System and PhEDEx as well as access to the align-ment and calibration database via Frontier. The Tier0 ran stably and mainlyunattended during the four weeks of the challenge with an uptime of 100%. Onlyscheduled operator inventions like changing the running reconstruction version,adjusting the rate of different physics channels or fixing trivial bugs in the Tier0framework helped to achieve all proposed metrics of CSA06.6 A.Scheurer,M.Ernst,A.Flossdorf,C.Hof,T.Kress,K.Rabbertz,G.Quast3.7 Data DistributionThe transfer rates to the Tier1 centres were expected at about 25% of theneeds in 2008. Table 1 gives an overview of the goals, threshold values andachieved transfer rates for the seven Tier1s as well as information about networkspeed and provided storage capacity. During the whole period of CSA06, morethan 1 PB was transferred among all participating tier sites. Figure 2 shows thecumulative transferred amount of data via the wide area network.Tier1 Goal Thres. 30day avg. 15day avg. Storage NetworkSite [MB/sec] [MB/sec] [MB/sec] [MB/sec] [TB] [Gb/sec]ASGC 15 7.5 17 23 48+(36) 2CNAF 25 12.5 26 37 40 10+(10)FNAL 50 25 68 98 700 11GridKa 25 12.5 23 28 40+(30) 10IN2P3 25 12.5 23 34 70 10PIC 10 5 22 22 50 1RAL 10 5 23 33 60+(20) 11Table 1: Intended and achieved transfer rates from CERN to the Tier1 centresas well as the provided storage space and network capacity. The parenthesisednumbers indicate installed components during CSA06.Figure 2: Cumulative data transferred during CSA06 between all participatingtier sites in TB.GES 2007 Challenges of the LHC Comp. Grid by the CMS experiment 73.7.1 Tier0 to Tier1 OperationsDuring the first weeks of CSA06, the transfer rate between CERN and theseven Tier1 centres did not exceed the desired target rate of 150 MB/sec. Theincrease of the Tier0 reconstruction rate resulted in much higher transfer ratesduring the second half of the challenge. As can be seen in table 1, all centresreached the goal within a 15 day average of the challenge, whereas five Tier1centres even succeeded during the whole 30 day period of CSA06. Figure 3 showsthe transfer rates between CERN and the Tier1 sites. Due to some databaseproblems, a higher amount of data was accumulated resulting in transfer ratesof over 300 MB/sec. Towards the end of the challenge, outgoing rates of morethan 350 MB/sec have been reached easily due to the higher Tier0 reconstructionrate. All shown transfer rates are averaged over 24 hours whereas the hourlyaverage even exceeded 650 MB/sec. Most Tier1 centres performed at a veryhigh level of quality. In case of GridKa, some stability problems with the initialversion of dCache needed frequent manual intervention and made a scheduledupgrade inescapable. The subsequent reconfiguration and tuning of the systemled to large inefficiencies during the last week of CSA06. Nevertheless, GridKafinished the challenge with only a few days of delay and record transfer rates ofover 440 MB/sec over more than 8 hours.Figure 3: Data transfer rate between Tier0 and Tier1 centres during the first 30days of CSA06 in MB/sec. The red line indicates the target rate for CSA06.3.7.2 Tier1 to Tier2 OperationsThe goals for CSA06 were reached by 20 out of the 27 participating Tier2sites, whereas one other centre reached the threshold anyhow. Figures 4 and 58 A.Scheurer,M.Ernst,A.Flossdorf,C.Hof,T.Kress,K.Rabbertz,G.Quastshow the data transfer rates between Tier1 and Tier2 centres and the transferquality for these transfers, respectively. Since the Tier2 sites are only expectedto transfer the data in bursts, the rates are not as continuous as they are for theTier1s. The transfer quality varied widely among the participating Tier2 centres.Figure 6 shows that the German Tier2 sites performed extremely well during thewhole period of CSA06 with successful transfers from GridKa to DESY at ratesexceeding 140 MB/sec.Figure 4: Data transfer rate between Tier1 and Tier2 centres during the first 30days of CSA06 in MB/sec.3.8 Data Re-ProcessingThe goal was to test the re-reconstruction of existing datasets with newcalibration constants from the offline database at an order of 100 000 eventsat each Tier1. Due to problems with the yet untested reconstruction code, notall Tier1 centres achieved this goal immediately. After skimming and runningover the resulting smaller data samples re-reconstruction succeeded at all Tier1centres.4 Monte Carlo ProductionIn preparation for CSA06, more than 67 million events had to be simulatedbeforehand and then were transferred to the Tier0. The grid-based Monte CarloGES 2007 Challenges of the LHC Comp. Grid by the CMS experiment 9Figure 5: Data transfer quality between Tier1 and Tier2 centres during the first30 days of CSA06.Figure 6: Data transfer quality for the DESY and Aachen Tier2 during CSA06and an example of transfer rates between GridKa and DESY.production started in mid July and was conducted by four operations teamsfrom the Tier2 centres CIEMAT, DESY/RWTH, INFN/Bari, and Universityof Wisconsin, Madison. Although primarily a task of the Tier2s, Tier1 centresaided in this exercise. The number of events produced at the German sites addsup to 14% of the whole CMS Monte Carlo Production.10 A.Scheurer,M.Ernst,A.Flossdorf,C.Hof,T.Kress,K.Rabbertz,G.Quast5 Physics AnalysesThe data distributed during CSA06 was not only used for pure testing, butwas at the same time used for physics studies by individual users. In addition, a”job robot” secured a base load of grid jobs being run at each centre. Here, theperformance of the resource broker was not sufficient to keep all available CPUsbusy.6 ConclusionsThe Computing, Software and Analysis challenge of the CMS collaborationperformed in the fall of the year 2006 demonstrated the base functionality ofthe grid infrastructure, grid services and the experimental software at a level of25% or more of the requirements foreseen for 2008. In particular the grid storageelements showed some fragility under heavy load and required substantial effortson the operational side. The German CMS sites, the Tier1 centre GridKa andthe federated Tier2 centre DESY/RWTH Aachen, successfully participated atall stages of the challenge. Much experience was gained with the execution ofcomplicated workflows on the grid infrastructure, and performance bottlenecksand instabilities were identified. Further service challenges at a larger scale willbe needed to guarantee the functionality of the whole system at the scale requiredfor the start-up of data-taking at the LHC.References1. CMS Collaboration, ”The Computing Project, Technical design report”,CERN/LHCC 2005-0232. J. Knobloch, L. Robertson, et al. ”LHC Computing Grid - Technical Design Re-port”. LCG-TDR-001 (2005). CERN-LHCC-2005-024. ISBN 92-9083-253-3.3. LHC and WLCG Collaboration ”Memorandum of Understanding for Col-laboration in the Deployment and Exploitation of the Worldwide LHCComputing Grid”. CERN-C-RRB-2005-01/Rev. http://lcg.web.cern.ch/LCG/C-RRB/MoU/WLCGMoU.pdf.4. CMS Collaboration ”CMS Computing, Software and Analysis Challenge in 2006(CSA06) Summary”. CMS NOTE 2007/006 (2007). CERN/LHCC 2007-010.LHCC-G-128.5. CMS PhEDEx Webpage. http://cmsdoc.cern.ch/cms/aprom/phedex/.6. Frontier Webpage. http://lynx.fnal.gov/ntier-wiki/.7. Squid Webpage. http://www.squid-cache.org/.8. CMS CRAB Webpage. http://cmsdoc.cern.ch/cms/ccs/wm/www/Crab/.9. Load Sharing Facility Webpage. http://www.platform.com/Products/Platform.LSF.Family/Platform.LSF/Home.htm.10. dCache Development Webpage. http://dcache.desy.de/.11. Maui Scheduler Webpage. http://mauischeduler.sourceforge.net/.12. Torque Resource Manager Webpage. http://www.clusterresources.com/pages/products/torque-resource-manager.php.13. Enabling Grids for E-Science Project Webpage. http://public.eu-egee.org/.

Challenges of the LHC Computing Grid by the CMS experiment

Scheurer, Armin

Ernst, Michael

Flossdorf, Alexander

Hof, Carsten

Kress, Thomas

Rabbertz, Klaus

Quast, Günter

Max Planck Society eDoc Server

https://pure.mpg.de/pubman/item/item_1786656_1/component/file_1786655/GES_paper85.pdf

Challenges of the LHC Computing Grid by the CMS experiment

Abstract

Similar works

Full text

Available Versions

MPG.PuRe

MPG.PuRe

Max Planck Society eDoc Server