The CMS experiment will soon produce a huge amount of data (a few PBytes per year) that will be distributed and stored in many computing centres spread across the countries participating in the collaboration. Data will be available

to the whole CMS physicists: this will be possible thanks to the services provided by supported Grids. CRAB is the CMS collaboration tool developed to allow physicists to access and analyze data stored over world-wide sites. It aims to simplify the data discovery process and the jobs creation, execution and monitoring tasks hiding the details related both to Grid infrastructures and CMS analysis framework. We will discuss the recent evolution of this tool from its standalone version up to the clientserver

architecture adopted for particularly challenging workload volumes and we will report the usage statistics collected from the CRAB community, involving so far almost 600 distinct users

Cinquilli, M.

Codispoti, G.

Fanfani, F.

Fanzago, F.

Farina, F.

Lacaprara, S.

Miccio, E.

Spiga, S.

Vaandering, E.

English

Scientific Open-access Literature Archive and Repository

DOI 10.1393/ncc/i2009-10399-2Colloquia: CSFI 2008IL NUOVO CIMENTO Vol. 32 C, N. 2 Marzo-Aprile 2009The Grid-distributed data analysis in CMSF. Fanzago(1)(2), F. Farina(1)(3), M. Cinquilli(4), G. Codispoti(5)(4),F. Fanfani(6), S. Lacaprara(7), E. Miccio(1)(3), S. Spiga(1)(4) andE. Vaandering(8)(1) CERN - Gene`ve, Switzerland(2) INFN-CNAF - Bologna, Italy(3) INFN, Sezione di Milano - Milano, Italy(4) INFN, Sezione di Perugia - Perugia, Italy(5) Universita` di Perugia - Perugia, Italy(6) Universita` di Bologna - Bologna, Italy(7) INFN, Sezione di Legnaro - Legnaro (PD), Italy(8) FNAL - Batavia, Illinois, USA(ricevuto il 22 Giugno 2009; pubblicato online il 10 Settembre 2009)Summary. — The CMS experiment will soon produce a huge amount of data (a fewPBytes per year) that will be distributed and stored in many computing centresspread across the countries participating in the collaboration. Data will be availableto the whole CMS physicists: this will be possible thanks to the services provided bysupported Grids. CRAB is the CMS collaboration tool developed to allow physiciststo access and analyze data stored over world-wide sites. It aims to simplify the datadiscovery process and the jobs creation, execution and monitoring tasks hiding thedetails related both to Grid infrastructures and CMS analysis framework. We willdiscuss the recent evolution of this tool from its standalone version up to the client-server architecture adopted for particularly challenging workload volumes and wewill report the usage statistics collected from the CRAB community, involving sofar almost 600 distinct users.PACS 07.05.Kf – Data analysis: algorithms and implementation; data management.1. – IntroductionThe CMS experiment is one of the four high-energy-physics experiment that willstart to collect data in the 2009 at LHC accelerator at CERN. The large quantity ofdata foreseen to be produced per year (approximately 2PB) and to be stored, the bigcomputing power required for analysis and data simulation and the need for data accessto all the physicists of the collaboration throughout the world induced CMS to base itscompunting model on the Grid concept.c© Societa` Italiana di Fisica 115116 F. FANZAGO, F. FARINA, M. CINQUILLI, ETC.The CMS computing model is geographically distributed, organized in a hierarchicalorder of regional computing centres: a single Tier-0 centre at CERN for prompt datareconstruction; few Tier-1 centres having custodial responsability for raw and recon-structed data and providing services for re-processing, calibration, skimming and otherdata-intensive analysis tasks; several Tier-2 centres where a fraction of reconstructeddata and high-level analysis object are replicated for user analysis providing also MonteCarlo simulation facilities.A CERN Analysis Facility (CAF) is also foreseen for low-latency activities requiringaccess to all raw data: it provides a rapid access to the entire CMS dataset for fastanalysis.The Grid infrastructure ensures remote-resources availability and assures remote-dataaccess to authorized users, belonging to the CMS virtual organization. Grid services areprovided by WorldWide LHC Computing Grid (WLCG) [1] and Open Science Grid(OSG) [2] to manage the distributed resources.The data analysis in a distributed environment is a complex task. It assumes to knowwhere data to analyze are located, how to access them, in which remote site the analysissoftware is installed and how to use the Grid infrastructure. In order to make the analysison remote resources feasible, the CMS Collaboration is developing some tools interfacedwith the already available Grid services, including a tool for data distribution amongtiers (Phedex), specific catalogues for remote data discovery (DBS) and a tool to helpusers to create, submit and manage analysis job in the Grid anvironment: CRAB (CMSremote Analysis Builder) [3, 4].2. – CRAB, the CMS Remote Analysis BuilderThe purpose of CRAB is to allow users not knowing Grid infrastructure to run theiranalysis code on data available only at remote sites as easily as in a local environment,hiding Grid architecture details.It consists of a userfriendly tool, written in python, to be installed on a UserInterface(the entry point for the Grid), where the software for the analysis (CMSSW) is available.The user has simply to develop its analysis code, to decide which data he want to analyzeand how to manage the produced analysis output. This means either to retrieve it inthe UI or to copy it to a remote storage element when the analysis job has finished. Theuser interacts with CRAB via a configuration file, by changing some parameter valuessuch as the name of the dataset to analyze, the total number of events, the name of itsCMSSW configuration file and by flagging the copy or retrieving of outputs and theirpublication.Once CRAB has read its configuration file, its flow can be factorized in the followingsteps:– Data discovery: it queries the DBS catalogue for data location. The query re-turns the list of sites where data are located and how they are distributed. Thisinformation is used in the job creation step.– Job creation: during this step, CRAB performs the packaging of the analysis usercode including executable, library and user data. Then it prepares the wrapper ofthe user analysis code, script to be executed on the remote farm. The wrapper setsup the correct running environment depending on the Grid middleware, launchesthe user executable and handles the output. CRAB creates also the jdl file, specificTHE GRID-DISTRIBUTED DATA ANALYSIS IN CMS 117script to submit job to the Grid whose requirements are the CMSSW version usedby user and the site name hosting the data.– Job submission and management: the submission and monitoring are performedusing BossLite (Batch Object Submission System). The management includes thecheck of jobs’ status, the possibility to kill and resubmit them. CRAB is able tosubmit jobs not only on Grid environment but also using local batch system asLSF(Load Sharing Facilities) for CAF.– Output retrieval and publication: When jobs have finished, CRAB takes chargeeither of retrieving analysis outputs into the UI or copying them to a remote SE,specified by user in the crab configuration file. In order to make the analysis resultsavailable to physics groups, the user can decide to publish them into a “local” DBS.So published data can be re-analysed via CRAB.CRAB provides the user with a simple command line, whose options are the CRABfunctionalities, i.e. crab -create; crab -submit.3. – CRAB evolution: the serverThe major effort of the development activity during the last year is focused to theclient-server implementation. The aim of this architecture is to automatize as muchas possible the interaction with the Grid by reducing the unnecessary human workloaddemanding all possible actions to the server side, improving also the scalability of thewhole system.The client is on the User Interface, while the server could be located somewhere overthe Grid. The CRAB client is directly used by physicists to perform the operationsinvolved in the task/job preparation and creation, such as the data discovery, the input-sandbox preparation, the job preparation (included the job splitting) and the requirementdefinition. Then the client sends the CRAB server a request completely transparent tothe user. The server manages: job submission to the Grid, automatic job monitoring,automatic job output retrieval, re-submission of failed jobs following particular rules fordifferent kinds of job failures, every specific command requested by the user as the jobkilling, and e-mail notification when the percentuage of finished jobs of a task reachsa specified level to the user. The communication between the client and the server isimplemented by using the web services technology, based on SOAP which is standardin the Grid service development community. It uses HTTP protocol and provides in-teroperability across application languages. The internal CRAB server architecture isbased on components implemented as independent agents communicating through anasynchronous and persistent message service based on a MySQL database. Each agenttakes charge of specific operations:– CommandManager: the endpoint of the SOAP services which handles commandssent by the CRAB client;– CrabWorker: component that performs direct job submission to the Grid;– TaskTracking: keeps all the general information about tasks under execution,polling it from the database by using threads;118 F. FANZAGO, F. FARINA, M. CINQUILLI, ETC.– Notification: notifies the user by an e-mail when his task has ended and the outputhas already been retrieved; it is also used to notify the server administrator aboutspecial warning situation;– TaskLifeManager: manages the task life on the server by cleaning ended tasks;– JobTracking: tracks the status of every job;– GetOutput: retrieves the output of ended jobs;– JobKiller: kills single or multiple jobs when required;– ErrorHandler: performs a basic error handling allowing jobs resubmission;– RSSFeeder: provides RSS channels which can be used to redirect information aboutthe server status;– AdminComponent: executes specific server maintenance operations;– HTTPFrontend: provide a web interface to monitor submitted tasks, resourcesusage, dataset accessed, and destination sites.Many of the above-listed components are implemented following a multithreading ap-proach, that allows to manage many tasks at the same time shortening the delay time fora single operation that has to be accomplished over many tasks. Two important entitiesin the CRAB server architecture are the WS-Delegation and a specific area on an exist-ing Storage Element. The WS-Delegation is a service for the user proxy delegation fromthe client to the server allowing the server to perform each Grid operation for a giventask with the corresponding user proxy. The SE allows to transfer the input/output-sandboxes between the User Interface and the Grid, working as a sort of drop-box area.The server interacts with any associated storage server independently of the protocolallowing to have a portable and scalable object, where the Storage Element hosting thejob sand-boxes is completely independent of the server.4. – CRAB usageCRAB has been in production since Spring 2004. The client-server architecture is bornat the end of 2007. CRAB was used by CMS physicists to analyze data for the PhysicsTecnical Design Report and during some LCG and CMS challenges such as, CSA06,CSA08, and CCRC08 with the aim to test the maturity of Grid infrastructure andCMS framework and to demostrate the feasibility of distributed analysis. Furthermore,CRAB is daily used to check the availability of remote resources, through an automatictool to spread job continuously over all published data. Currently about 100 users perday submit through CRAB analysis jobs, reaching a peak of 70000 jobs per day andmantaining an average of about 10000 jobs per day.5. – ConclusionCRAB is the official distributed analysis tool of CMS. The aim of CRAB is to simplifythe creation and the management of analysis jobs in the Grid environment. It was bornas simple tool to be installed in the UI, but the need for scalability and robustness toface the foreseen number of jobs going to be submitted when real data are availableTHE GRID-DISTRIBUTED DATA ANALYSIS IN CMS 119has driven its evolution to a client-server architecture. Currently, CRAB is able toperform three main kinds of job submission, totally transparent to the user: the directsubmission to the Grid interacting directly with different middlewares (LCG and OSG);the direct submission to local resources and relative queues, using the batch system; thesubmission to the Grid using the CRAB server as a layer responsible of the interactionwith the middleware and the task/job management.CRAB is being extensively using by the CMS community, that provides ideas and usecases for new functionalities and improvements.REFERENCES[1] LHC Computing Grid (LCG), Web Page, http://lcg.web.cern.ch/LCG/ and LCGComputing Grid - Technical Design Report, LCG-TDR-001 CERN/LHCC 2005-024 (2005).[2] OSG Web Page, http:/opensciencegrid.org/.[3] Codispoti G. et al., CRAB: a CMS Application for distributed analysis, preprint 2008Nuclear Science Symposium.[4] Spiga D. et al., The CMS Remote Analysis Builder (CRAB), in Nucl. Phys. B, 4873 (2007)580.

The Grid-distributed data analysis in CMS

http://eprints.bice.rm.cnr.it/16524/1/ncc9395.pdf

The Grid-distributed data analysis in CMS

Abstract

Similar works

Full text

Available Versions

Scientific Open-access Literature Archive and Repository