Approaching LHC data taking, the CMS experiment is deploying, commissioning and operating the building tools of its grid-based computing infrastructure. The commissioning program includes testing, deployment and operation of various storage solutions to support the computing workflows of the experiment. Recently, some of the Tier-1 and Tier-2 centers supporting the collaboration have started to deploy StoRM based storage systems. These are POSIX-based disk storage systems on top of which StoRM implements the Storage Resource Manager (SRM) version 2 interface allowing for a standard-based access from the Grid. In this notes we briefly describe the experience so far achieved at the CNAF Tier-1 center and at the IFCA Tier-2 center

Bonacorsi, D

Cabrillo Bartolome, I

González-Caballero, I

Matorras, F

Sartirana, A

BONACORSI, DANIELE

I. Cabrillo Bartolomé

I. González Caballero

F. Matorras

A. Sartirana

Archivio istituzionale della ricerca - Alma Mater Studiorum Università di Bologna

The CMS experiment workflows on StoRM based storage at Tier-1 and Tier-2 centers

CERN Document Server

Available on CMS information server CMS CR -2009/080The Compact Muon Solenoid ExperimentMailing address: CMS CERN, CH-1211 GENEVA 23, SwitzerlandConference Report07 May 2009 (v2, 14 May 2009)The CMS experiment workflows on StoRM basedstorage at Tier-1 and Tier-2 centersD Bonacorsi, I Cabrillo Bartolome´, I Gonza´lez Caballero, F Matorras and A SartiranaAbstractApproaching LHC data taking, the CMS experiment is deploying, commissioning and operating thebuilding tools of its grid-based computing infrastructure. The commissioning program includes test-ing, deployment and operation of various storage solutions to support the computing workflows ofthe experiment. Recently, some of the Tier-1 and Tier-2 centers supporting the collaboration havestarted to deploy StoRM based storage systems. These are POSIX-based disk storage systems on topof which StoRM implements the Storage Resource Manager (SRM) version 2 interface allowing for astandard-based access from the Grid. In this notes we briefly describe the experience so far achievedat the CNAF Tier-1 center and at the IFCA Tier-2 center.Presented at CHEP - 17th International Conference on Computing in High Energy and Nuclear Physics.,21 - 27March 2009 ,Prague,Czech Republic,15/05/2009The CMS experiment workows on StoRM basedstorage at Tier-1 and Tier-2 centersD Bonacorsi1, I Cabrillo Bartolome2, I Gonzalez Caballero3, FMatorras2 and A Sartirana41Univ. of Bologna / INFN, Italy2IFCA, Univ. of Cantabria-CSIC, Spain3Univ. of Oviedo, Spain4LLR, Ecole Polytechnique, FranceE-mail: sartiran@llr.in2p3.itAbstract.Approaching LHC data taking, the CMS experiment is deploying, commissioning andoperating the building tools of its grid-based computing infrastructure. The commissioningprogram includes testing, deployment and operation of various storage solutions to supportthe computing workows of the experiment. Recently, some of the Tier-1 and Tier-2 centerssupporting the collaboration have started to deploy StoRM based storage systems. These arePOSIX-based disk storage systems on top of which StoRM implements the Storage ResourceManager (SRM) version 2 interface allowing for a standard-based access from the Grid. In thisnotes we briey describe the experience so far achieved at the CNAF Tier-1 center and at theIFCA Tier-2 center.1. IntroductionIn preparation of LHC data taking, the CMS experiment has exercised its computing model inperiodic challenges, with increasing scale and complexity, as well as in daily production leveloperations.The CMS computing system relies on the tools of the Worldwide LHC Computing Gridproject [2]. On top of them CMS builds its own experiment-specic applications: dataplacement and transfer system, distributed analysis and MonteCarlo (MC) production tools,data bookkeeping and location system, etc. The system is based on a model of \Tiers" ofresources (Tier-0, Tier-1s, Tier-2s) with specic functionalities [1, 2, 3].The Tier-0, located at CERN laboratory, provides real time reconstruction, tape archivingand distribution to Tier-1s of detector data. The Tier-1s centers, 7 national level centers, providesecondary archive and re-reconstructions of data, serve real data to Tier-2s and import MC data.The Tier-2s, about 50 smaller centers in the world, generate MC samples and provide resourcesand tools for Physics analysis.A key role in the setup of the site's activities is played by the deployment and the operationof storage systems. A number of dierent solutions have been adopted and widely tested by thevarious sites supporting the collaboration: DPM, dCache, Castor, etc. Recently, an increasingnumber of sites have moved to adopt StoRM based storage systems [5, 4]. During the lastchallenges this solution has been thoroughly tested with excellent results, and is now stablydeployed in everyday operations by many Tier-2s sites and one Tier-1.In this note we will report about the experience achieved with StoRM storage at the CNAFTier-1 center and the IFCA Tier-2 center. In section 2 we quickly review the basics of StoRMstorage and of IBM General Parallel File System (GPFS), section 3 briey describe the statusof integration between StoRM and the CMS-specic applications, the experience achieved withStoRM at CNAF and IFCA is described in section 5 and 6 respectively.2. StoRM SRM SystemStoRM is a storage resource manager (SRM) for disk based storage systems implementing theSRM interface version 2.2 [5].The SRM interface allows the Grid applications to have a standard-based access to the diskstorage without any need of knowing the details of the underlying storage technology. Allrequests of reading/writing les, as well as executing namespace operations (listing, creatingand removing directories, etc), are managed by the SRM service in a transparent way. Grid-level authentication is provided by standard GSI interface. The direct interaction with thestorage system is performed by the SRM which takes care of extracting the les physical names,reserving the storage space, implementing the ownership policies, etc.As a SRM service, StoRM works on top of any standard POSIX le system with ACL support,like GPFS[6], XFS[7] and ext3[8]. It supports GridFTP, ro or le as access protocols (g. 1).Access is provided by standard POSIX operations, without interacting with any external servicethat emulates data access, with the result of improving the performance when the underlyingle system is ecient. StoRM hence takes advantage from high performance parallel and clusterle systems. ACLs provided by the underlying le systems are used by StoRM to implementthe security model.StoRM has a multilayer architecture, g. 2, built of two main components: the front-end andthe back-end. The front-end exposes the SRM web service interface, manages user authenticationand stores the requests data into the database. The back-end executes all synchronous andasynchronous SRM functionalities, takes care of le and space meta-data management, enforcesauthorization permissions on les and interacts with le transfer services.Thanks to a plug-in mechanism, the back-end is able to exploit the advanced functionalitiesprovided by the le system (for example by GPFS and XFS) in order to accomplish spaceFigure 1. Access to StoRMstorage by grid applications.Figure 2. StoRM architecture.reservation requests. The plug-in mechanism keeps the back-end logic decoupled from thewrapper component that provides le system support, which considerably simplify adding newsupport for dierent le systems.2.1. GPFSIt is worth spending few words of introduction to the IBM General Parallel File System sincethe StoRM storage solutions which are presented in this note are based on this technology.GPFS is a le system designed for high-performance parallel access on disk pool storage[6].GPFS can build and serve a single le system containing up to Petabytes of data on dedicatedhardware in a Storage Area Network (SAN) conguration. Moreover, GPFS can providea software simulation of a SAN storage by the Network Shared Disk (NSD) functionality.Exploiting NSD, GPFS storage system can thus extend and serve data with high performancesto wide pools of nodes (such as big farms) which cannot be included in the SAN Fiber Channelnetwork.3. StoRM and CMS WorkowsThe deployment of StoRM based storage at the sites that support the CMS activities obviouslyposes some compatibility issues which have to be addressed in order to be able to perform all theCMS typical workows. Considerable eort has been spent on this topic by the administratorsof the concerned sites, by the developers of CMS applications and by the StoRM developmentteam. In this section there is a brief description of the status of the integration between StoRMstorage systems and the CMS computing.CMS-specic applications are, in most cases, built to be compliant with the SRMv2 standards,which remarkably simplies their integration with StoRM based storage.For example, PhEDEx is the CMS data transfer and placement system which manages, atVO level, all the data placement requests to the CMS sites, it submits the le transfers requestto the Grid services and checks that the les are correctly copied at sites. It also manages datadeletion from sites. PhEDEx relies on agents located at sites for performing all local operations:les removal, le validation after transfers, etc. The site agents can be congured in order touse general SRMv2 clients while performing such local operations. In particular the SRMv2client used by PhEDEx are the native dCache ones; dCache is, like StoRM, a storage managersystem which is deployed in many Tier-1 and Tier-2 sites supporting CMS, and its clients arefully compatible with all SRMv2 standard based storage systems.For what concerns the staging out to local storage of the output of production jobs, it ispossible to set it up in such a way that it is performed by using the same dCache clients. Thissolution is clean and quite easy to implement but may not be optimal for this use case. dCacheSRM clients use, in fact, the GridFTP protocol to write les. GridFTP doors are usually limitedin number and the GridFTP server is often optimized for long distance WAN transfers: largeallocated buer, etc. A better solution is to write les directly with the POSIX 'cp' command.In order to do this in a clean way, the command should be enclosed by a PrepareToPut anda PutDone SRMv2 calls. At the moment such operation is not available in the framework ofCMS applications. Despite this limitation, most of SToRM T2s are successfully using POSIXstageout exploiting some locally implemented workarounds.For what concerns read access to local les from both production and analysis jobs, thiscan be performed by direct POSIX interaction with the GPFS le system. The physical pathon GPFS can be provided to the jobs by the Trivial File Catalogue (TFC). TFC is a localconguration le, used by CMS jobs, which contains the rules for the logical-to-physical lename mapping. Each site has its own TFC and can modify it in such a way to best t thelocal storage conguration. This turned out to be a very ecient solution leading to remarkableperformances and stability.As the use of StoRM would spread over sites, an eort to further integrate CMS applicationswith StoRM based storage systems would be required.4. StoRM at a Tier-1 center: the CNAF caseINFN-CNAF is a multi-VO Tier-1 located in Bologna, Italy. CNAF supports all four LHCexperiments as well as some non-LHC collaborations (BaBar, CDF, Argo, Virgo, etc). Moreoverit contributes to various European development projects like LCG, EGEE, INFN Grid, etc.4.1. Storage setupCNAF hosts a hybrid storage system with a Castor based tape back-end storage and aStoRM/GPFS based disk-only storage. The center hosts 7 GPFS clusters, version 3.2.1-4,serving more than 1 PB of storage to approximately 600 nodes: worker nodes, user interfaces,GridFTP servers and GPFS servers. Disk storage is connected by a 2-4Gb Fiber Channel SANsystem. CMS has a dedicated StoRM, version 1.3.20-03, instance with 2 front-end machinesand one back-end machine, a dedicated 190TB of GPFS storage and a pool of three dedicatedGridFTP servers. Presently, all GridFTP servers are Scientic Linux CERN 4 (SLC4) 32bitserver accessing to the GPFS storage via LAN connection. This conguration implies a limitationin the maximum rate per servers: 1Gb/s maximum inbound/outbound aggregate; moreover,the GridFTP server on 32bit OS has shown to have memory allocation issues. A dierentconguration, with 64bit SLC4 GridFTP servers directly accessing the storage via SAN FiberChannel will soon be deployed.4.2. Experience achieved and future plansGPFS storage has been in use by CMS at CNAF since the end of 2007 hosting the disk-onlyarea for the unmerged MC les. These are les which are created by the MC production jobs,they are temporarily stored on disk, waiting for being merged, by other jobs, in larger les whichwill eventually go on tape storage. This solution showed to be remarkably stable. The StoRMendpoint was, instead, rst deployed on February 2008 and tested for the rst time duringCCRC08 Phase I and II [10]. After some initial rump up during the challenge, the endpoint hasbeen constantly exercised with the CMS LoadTest infrastructure (g. 3, g 4). Since Apr 2009the StoRM/GPFS storage is also employed in the continuous reconstruction exercise which isperformed by CMS in order to keep Tier-1s lled with processing jobs and ready in sight of datataking.Figure 3. StoRM import transfervolume to CNAF, last year up toMarch 2009Figure 4. StoRM import transfervolume from CNAF, last year up toMarch 2009CNAF is also working to deploy a full disk + tape back end solution based on StoRM, GPFSand TSM[9]. Once a, rst, fully operational StoRM/TSM solution for tape backend storage willbe deployed, CMS operations at CNAF will consider to start moving from Castor to StoRMstorage. Detailed plans for such migration are still under evaluation.5. StoRM at a Tier-2 centre: the IFCA caseIFCA is a multi-VO Tier-2 located in Santander, supporting CMS in LHC and other non-HEP communities: Plank in Astrophysics, statistical Physics, Biomedicine, etc. It also hostsseveral GRID computing projects like NGI-ES, DORII, EGEE, EGI, EUFORIA, GRID-CSICand INTEUGRID.5.1. SetupIFCA is currently running StoRM version 1.3.20-03. The present conguration of the endpointconsists of a unique node hosting all StoRM services: front-end, back-end and MySQL database.High performance WAN transfers are served by a pool of 4 GridFTP servers. The underlyingPOSIX le system is provided by a GPFS cluster. The GPFS Storage Network is deployed on topof a private LAN. StoRM and the GridFTP servers must have access to both networks. With thisconguration all the farm has access to GPFS through the usual POSIX linux commands (cp,rm, mv, etc) and the GPFS storage can be used as any other local le system. This simpliesthe access to the data on the Worker Nodes and improves the performances and availabilitycompared to other storage systems that need to implement other access protocols.5.2. Experience with StoRMThe installation of StoRM at IFCA was proposed in order to solve the large request for datatransfer and availability of some projects, in particular of CMS. The StoRM endpoint IFCA wasdeployed in production for the rst time on March 2008. Since then the endpoint supportedall the IFCA activity with good performance and availability. In g. 5 and 6 you can see thevolume of data transferred with the StoRM endpoint, inbound and outbound, in the last 52weeks up to March 2009. From the administration point of view StoRM showed to be easy toinstall, the procedure is quite straightforward by simply following the INFN-GRID instructions,and easy to maintain.Figure 5. StoRM import transfervolume to IFCA, one year up toMarch 2009Figure 6. StoRM export transfervolume from IFCA, one year up toMarch 20096. Summary and OutlookStoRM storage system, despite being one of the youngest which showed up in the frameworkof Computing in High Energy Physics, has already spread in many sites and was widely testedin the operation of LHC experiments. Many CMS sites have recently turned to operate StoRMstorage systems with good results. At the moment, the system works with a good level ofintegration with the CMS computing framework but some further eort will be needed in orderto make StoRM more and more compatible with the standard CMS workows.StoRM has been adopted by the IFCA Tier-2 as its main storage system and by the CNAFTier-1 in a hybrid storage setup together with Castor. The experience so far achieved in thesetwo sites, both from challenges and from daily operations, is very positive and promising in thesight of LHC data taking.References[1] C.Grandi, D.Stickland, L.Taylor et al., \The CMS Computing Model" CERN-LHCC-2004-035/G-083 (2004)[2] I. Bird et al., \LHC computing Grid. Technical design report", CERN-LHCC-2005-024 (2005)[3] The CMS Collaboration \CMS Computing Technical Design Report", CERN-LHCC-2005-023,(2005)[4] A. Carbone et al., \Performance studies of the StoRM Storage Resource Manager,"http://www.slac.stanford.edu/spires/nd/hep/www?irn=8199450SPIRES entry Proceedings of ThirdIEEE International Conference on e-Science and Grid Computing[5] R. Zappi, L. Magnoni, F. Donno and A. Ghiselli, \StoRM: Grid middleware for disk resource management,"http://www.slac.stanford.edu/spires/nd/hep/www?irn=7874316SPIRES entry[6] Schmuck, Frank; Roger Haskin (January 2002). \GPFS: A Shared-Disk File System for Large ComputingClusters" Proceedings of the FAST'02 Conference on File and Storage Technologies: 231-244, Monterey,California, USA: USENIX. Retrieved on 2008-01-18[7] http://www.sgi.com/products/storage/software/xfs.html[8] \Journaling the Linux ext2fs Filesystem" Stephen C. Tweedie (May 1998). Proceedings of the 4th AnnualLinuxExpo, Durham, NC. http://jamesthornton.com/hotlist/linux-lesystems/ext3-journal-design.pdf.Retrieved on 2007-06-23.[9] http://www-01.ibm.com/software/tivoli/products/storage-mgr/[10] P. Mendez Lorenzo, A Sciaba, S. Campana et al. \The WLCG common computing readiness challenge:CCRC`08" 3rd EGEE User Forum, Clermont-Ferrand, France, 11 - 14 Feb 2008

17th International Conference on Computing in High Energy and Nuclear Physics, CHEP 2009. 21 March 2009 through 27 March 2009. Pragu

The CMS experiment workflows on StoRM based storage at Tier-1 and Tier-2 centers

Abstract

Similar works

Full text

Available Versions

Archivio istituzionale della ricerca - Alma Mater Studiorum Università di Bologna

CERN Document Server

Repositorio Institucional de la Universidad de Oviedo