In this paper we will describe the Storage Infrastructure of the INFN-CNAF Tier-1, used to store data of High Energy Physics experiments, in particular those operating at the Large Hadron Collider

Cavalli, A.

dell'Agnello, L.

Gregori, D.

Magnoni, L.

Martelli, B.

Peco, G.

Prosperini, A.

Ricci, P. P.

Ronchieri, E.

Sapunenko, V.

Vagnoni, V.

Zappi, R.

English

Scientific Open-access Literature Archive and Repository

DOI 10.1393/ncc/i2009-10413-9Colloquia: CSFI 2008IL NUOVO CIMENTO Vol. 32 C, N. 2 Marzo-Aprile 2009Storage Infrastructure at the INFN LHC Tier-1A. Cavalli(1), L. dell’Agnello(1), D. Gregori(1), L. Magnoni(1),B. Martelli(1), G. Peco(2), A. Prosperini(1), P. P. Ricci(1), E. Ronchieri(1),V. Sapunenko(1), V. Vagnoni(2) and R. Zappi(1)(1) INFN-CNAF - viale Berti-Pichat 6/2, 40127 Bologna, Italy(2) INFN, Sezione di Bologna - via Irnerio 46, 40126 Bologna, Italy(ricevuto il 4 Agosto 2009; pubblicato online il 6 Ottobre 2009)Summary. — In this paper we will describe the Storage Infrastructure of theINFN-CNAF Tier-1, used to store data of High Energy Physics experiments, inparticular those operating at the Large Hadron Collider.PACS 89.20.Ff – Computer science and technology.1. – IntroductionThe growth of resource needs for the LHC experiments requires larger and largerquantities of data storage and increasing perfomance demands. In the framework ofWLCG, all the Tier-1 sites will have to provide different access types to the storagesystems, both disk resources for faster access and tape resources for long-term datacustodial. At the INFN-CNAF Tier-1, all storage resources are organized in a StorageArea Network (SAN) and served by Linux machines used as disk and tape servers; toallow for redundancy, all the servers and the storage resources have two interconnectionsto the SAN. In the next sections we will describe our infrastructure in terms of hardwareand software in use.2. – StoRMThe StoRM service [1,2] is a storage resource manager for generic disk based storagesystems implementing the Storage Resource Manager (SRM) interface version 2.2 [3].StoRM is designed to support guaranteed space reservation and direct access (nativePOSIX I/O call), as well as other standard Grid access protocols. It separates the datamanagement layer from the underlying storage systems characteristics.StoRM takes advantage from high-performance parallel file systems like GPFS (fromIBM) and Lustre (from Sun Microsystems) but it is able to manage data on any otherc© Societa` Italiana di Fisica 243244 A. CAVALLI, L. DELL’AGNELLO, D. GREGORI, ETC.standard POSIX file system through a driver mechanism. Cluster file systems allow sev-eral disk LUNs attached to multiple storage servers to be configured as a single file sys-tem, providing transparent parallel access to storage devices, while maintaining standardUNIX file system semantics. StoRM is able to leverage the advantages resulting from acluster approach in a Grid environment, enabling data intensive applications to directlyaccess data on the storage resource without interacting with any other transfer services.A modular architecture decouples the StoRM logics from the different file systemssupported, and plug-in mechanisms allow an easy integration of new file systems. Withthis approach, our data centre is able to choose the preferred underlying storage systemmaintaining the same SRM service.Another important characteristic of StoRM is the capability to identify the physicallocation of a requested data without querying any database service, but simply evaluatinga configuration file, an XML schema that describes the storage namespace and inputparameters as the logical identifier and SRM attributes. StoRM relies on the underlyingfile system structure to identify the physical location of the data file.Authorization on file and space access is another important driving feature of StoRM.To enforce permissions on files and directories, StoRM is able to use the ACL supportprovided by the underlying file systems. The resulting security model is highly config-urable in order to satisfy the requirement coming from heterogeneous scenarios, such asthe case of High Energy Physics (HEP) and Economics and Financial applications.StoRM is an open source project and it is distributed together with the gLite middle-ware within the INFN Grid release [4], that collects Grid component packages with toolsto automatically install and configure services on Scientific Linux based hosts. StoRM iscurrently adopted in the context of the World-wide LHC Computing Grid infrastructure(WLCG) in various data centres, such as the Italian Tier-1 at the INFN-CNAF institute,and different Tier-2 centres across the world.3. – CastorCASTOR [5] stands for CERN Advanced STORage manager and is a hierarchicalstorage management (HSM) system developed at CERN and used to store physics pro-duction files and user files. At our Tier-1, CASTOR has been our choice for the D0T1(tape based storage with disk stage area as frontend) implementation for the last yearswith effective results. Data is copied from the user resource to the CASTOR front-end(disk servers with connected disk space) and then subsequently copied to the back-end(tape servers). Data buffering allows the system to optimize the disk-to-tape copy (mi-gration process). Currently, we have 40 disk servers as front-end and each of them hasfive or six file systems. Typical size is comprised between 1.5 to 2TB for each file sys-tem, and both XFS and EXT3 are used. Disk servers are connected to the SAN usingfull redundancy on FC 2Gb/s (on latest machines 4Gb/s) connections, with a hardwaredual controller and a software Qlogic SANsurfer Path Failover (or in other cases vendorspecific software). As CASTOR back-end we use tape servers (each one with dedicatedtape drive) which are connected via FC to 6 LTO2 and 10 9940B tape drives which arelocated inside a single STK L5500 silo. The silo itself is partitioned with 2 form-factorslots. For LTO2 tapes there are 2000 slots and the other 3500 slots are for 9940B tapes(5500 slots in total). Both the technologies use 200GB cartridges, so the total capacity isroughly 1.1 PB (non-compressed). The library operations are managed with a dedicatedSun Blade v100 machine with 2 internal IDE disks with software RAID1 running ACSLS7.0 and Solaris OS.STORAGE INFRASTRUCTURE AT THE INFN LHC TIER-1 245When accessing a file that is not already in the disk staging area, data is obtainedfrom tape and copied to disk (recall process). Files can be stored, listed, retrieved andaccessed in CASTOR using command line tools or applications built on top of differentdata transfer protocols like RFIO (Remote File IO), GridFTP and XROOTD. CASTORprovides a UNIX like directory hierarchy of file names. The CASTOR namespace can beviewed and manipulated only through CASTOR client commands and library calls andit is provided by a Name Server. Currently, we are running CASTOR version v2.1.6-12.The Core services run on machines with SCSI disks, hardware RAID 1 and redundantpower supplies. We have distributed CASTOR Core Services basically on three differentmachines (Scheduler, Stager and Name Server) and on a fourth machine we run the Dis-tributed Logging Facility (DLF) for centralizing log messages and accounting information.CASTOR is driven by a database-centric architecture. A great number of componentsare stateless and the code is interfaced with Oracle relational database. At CNAF CAS-TOR is used by the four LHC experiments (ALICE, CMS, ATLAS, LHCb) and by eightother experiments (LVD, ARGO, VIRGO, AMS, PAMELA, MAGIC, BABAR, CDF).Even if CASTOR is mainly used for the management of the tape back-end, some ex-periments use CASTOR as pure disk, without tape migration. CASTOR uses the LSFscheduler [6] to determine the best candidate resource (file system) for a castor job. Thedecision depends on the load of disk server and on the number of assigned slots per diskserver. In fact, LSF slots are dedicated to each disk server (from 30 to 450) and canbe modified in case of need. The SRM (v.2) interface to CASTOR, for Grid users, isprovided by two end-points. One is dedicated only to the tape service class, while theother to the disk-only service class.4. – OracleAn Oracle database service is deployed for relational data storing/retrieving. Thedatabase service is an infrastructural layer which lies beneath several software systemsranging from experiments metadata archiving/retrieving systems, to World-wide LHCComputing Grid (WLCG) services and monitoring services. A non-exhaustive list ofsuch database-based systems comprises: detector conditions databases (CondDB), eventTAG collections, calibration databases, LFC (LCG File Catalog), FTS (File Transfer Ser-vice), VOMS (Virtual Organization Management Service), CASTOR (CERN AdvancedSTORage Manager), LEMON (LHC Era MONitoring).Due to its criticality, the Oracle database services is deployed in a fully redundantmanner implementing High Availability (HA) techniques at various levels:– At the hardware storage level, RAID controllers and Storage Area Networks providefull redundancy and failover of each LUN seen by each server.– At the software storage level, ASM (the Oracle Advanced Storage Manager) imple-ments striping and mirroring of Oracle database blocks across a group of LUNs.– At the database level, the Oracle Real Application Cluster (RAC) technology allowsto share a database amongst multiple servers, implementing load balancing andfailover.– At the WAN level, database replication via Oracle Streams is deployed. Thistechnique allows to ship Oracle data from CERN to CNAF, providing a disasterrecovery and load balancing solution on the wide area network.246 A. CAVALLI, L. DELL’AGNELLO, D. GREGORI, ETC.Database on-line backup is performed through Oracle Recovery Manager (RMAN)with a central catalog set-up. In the near future, we foresee to integrate the Oraclebackup service with our TSM infrastructure by means of the RMAN-TSM plug-in whichallows sending a backup copy to tape directly from RMAN.Presently the Oracle database service is composed by 32 dual-core servers organizedin 7 clusters, serving a total amount of 10TB of Fibre Channel storage plus 10 TB ofSATA disks. Thanks to its fully HA deployment, CNAF database service has achievedan availability rate of 98.7% in 2007.5. – GPFS-TSMCurrently, we have about 2600 TB of raw disk storage online and the space will beincreased by roughly 90% during the next year. The great majority of the disks are SATAones, due to a low-cost/good-performance compromise, but for specific applications (e.g.,database ones) which require a higher level of reliability and higher performances in therandom I/O access, native Fibre Channel disks are employed. The choice of using FibreChannel and SAN technology for all the disk hardware at our Tier-1 was driven by thefollowing considerations:– Disk-servers implement a so-called no “Single Point of Failure” (SPoF) system,where each component of the storage system is redundant.– The SAN gives the best flexibility since we can dynamically assign new volumes ordisk storage boxes to disk-servers without stopping the service.– LAN free systems both for archiving and backup purpose to the tape facilities arepossible.The General Parallel File System (GPFS) [7] is the file system adopted to store dataon disks at CNAF. GPFS is a world-wide distributed software developed by IBM, whichprovides a Parallel File System implementation. The idea of our implementation ofGPFS is to provide a fast and reliable service, without single point of failures, parallelfilesystem with direct access (POSIX file protocol) from the Farm worker nodes (i.e. the“clients”) using block level I/O interface over standard Ethernet network. In such animplementation, the clients do not need to have direct access to the SAN, they insteadcontact the GPFS Network Shared Disk (NSD) disk servers using the LAN and the diskservers provide all the I/O over the SAN and the storage boxes layers.Since GPFS works in a cluster configuration, with an opportune SAN hardware atrue higly available system is possible (disk server failures just decrease the maximumtheoretical bandwidth, but the file system remains always avaliable to the clients) and asingle “big file system” for each Virtual Organization (VO) is possible, which is stronglypreferred by users.Furthermore, from GPFS v.3.2, the concept of “external storage pool” was introduced.The external storage pool extends the use of a policy driven migration/recall system toa tape storage back-end such as Tivoli Storage Manager (TSM) [8], hence allowing theimplementation of a complete HSM system based on GPFS and TSM.STORAGE INFRASTRUCTURE AT THE INFN LHC TIER-1 247REFERENCES[1] Magnoni L., Zappi R. and Ghiselli A., StoRM: a Flexible Solution for Storage ResourceManager in Grid, in Proceedings of the IEEE 2008 Nuclear Science Symposium (NSS-MIC2008), 2008, Dresden, Germany.[2] Carbone A., Dell’Agnello L., Forti A., Ghiselli A., Lanciotti E., MagnoniL., Mazzucato M., Santinelli R., Sapunenko V., Vagnoni V. and Zappi R.,Performance studies of the StoRM Storage Resource Manager, in Proceedings of the 3rdIEEE International Conference on e-Science and Grid computing (eScience2007), 2007,Bangalore, India.[3] Sim A. and Shoshani A., The Storage Resource Manager Interface Specification Version2.2, Grid Final Documents, 2008, OGSA Data Working Group, Open Grid Forum (OGF),Available [ONLINE] at http://www.ogf.org/documents/GFD.129.pdf.[4] INFNGRID Release Wiki, Available [ONLINE] at http://igrelease.forge.cnaf.infn.it/.[5] For more documentation about the CASTOR software and RFIO protocol see [ONLINE]http://castor.web.cern.ch/castor/.[6] Load Sharing Facility (LSF) is a commercial computer software job scheduler developed byPlatform Computing. More documentation is available [ONLINE] at http://www.platform.com/Products/platform-lsf/.[7] IBM GPFS product, Available [ONLINE] at: http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/topic/com.ibm.cluster.gpfs.doc/gpfsbooks.html/.[8] IBM Tivoli product, IBM Tivoli Storage Management Concepts, IBM Redbooks Series,SG24-4877.

Storage Infrastructure at the INFN LHC Tier-1

http://eprints.bice.rm.cnr.it/16566/1/ncc9418.pdf

Storage Infrastructure at the INFN LHC Tier-1

Abstract

Similar works

Full text

Available Versions

Scientific Open-access Literature Archive and Repository