Presentation at 40th MICE Collaboration Meeting (CM40), 26th-29th October 2014. https://indico.cern.ch/event/331481/timetable/ Slides titled "Grid Status" 

Nebrensky, J

Brunel University Research Archive

Grid StatusHenry NebrenskyBrunel UniversityHenry Nebrensky – MICE CM40 – 26 October 2014 1Tears of MICEMLCR RAL CaSTOR TapeData Mover0GDARECO 1Grid SEmicestoreHTTPusersTearRAL T1ImperialBrunelRAL PPDHenry Nebrensky – MICE CM40 – 27 October 2014Data flow has a similar architecture to the LHC experimentsThe Data Mover and Batch Reprocessing frameworks are working.The GDA tools exists, but we don’t yet know how to manage the process. MonteCarlo Production has the process defined but implementation barely starting; expect significant code sharing with the Reco. 2MCGrid SE 2GDA GlasgowData Mover Permanent storage of data from the MICE DAQ.• Initial transfer/“File Compactor” still triggered manually (moveFiles script)• Autonomous process - “Data Mover” – then makes copy at RAL Tier 1 for tape archival (RAL PPD is now skipped).Henry Nebrensky – MICE CM40 – 27 October 2014• Separate agents then make copies at Imperial and BrunelMLCR DataMover PC has been moved to SL6, may move to state tracking in MySQL DB, other tweaks. Considering multi-threading if transfer rates to Castor are too poor.3MICE Raw DataRaw data on Grid, for up to June 2014 run inclusive • Data validation up to last June (run 5842)• All data tarballs valid (internal checksums OK)• All data tarballs have two copies on tape at Tier 1 Castor• All data tarballs have overall checksums match Henry Nebrensky – MICE CM40 – 27 October 2014between Castor and MLCR copy• Copies of data at (RAL PPD), Imperial College (web) and Brunel (Grid/WebDAV)4CVMFSCVMFS is• a read-only filesystem based on HTTP; uses caching to give (usable) global coverage• the master copy, Stratum-0, is at the RAL Tier1• installed on Grid clusters at Brunel, Imperial and RAL PPD and Tier 1Henry Nebrensky – MICE CM40 – 27 October 2014• MAUS is compiled and built at Imperial, then the binaries moved to the Stratum-0 and replicated across the Grid• Now a manual, interactive process so we get more feedback during the install• Have so far installed MAUS-v0.6.0, 0.8.1, 0.8.5, 0.9.15Offline Reco MAUS installed at RAL Tier1 for Offline Reco • automated process waiting for new raw data files• when a new data file appears, makes Offline Reco and run-specific Monte-Carlo job using latest approved MAUS • uses fast-response queue at RAL Tier1Henry Nebrensky – MICE CM40 – 27 October 2014• automatically runs MAUS jobs, creates output tarballs, saves them to Grid disk and tape :)Steps tested in principle during EMR run. Needs automation (dongle) and exercising of the system. 6Batch Reprocessing A manually-triggered process, in response to a particular MAUS release becoming available via CVMFS:• makes Offline Reco and run-specific Monte-Carlo job using that MAUS release for every run in a specified Step. Henry Nebrensky – MICE CM40 – 27 October 2014• uses Tier2 sites across the UK• automatically runs MAUS jobs, creates output tarballs, saves them to Grid disk and tape :)• Demonstrated last autumn with MAUS-v0.6.0 • Running with MAUS-v0.9.1 right now, after MAUS memory leaks resolved7MC Production The procedure is conceptually similar to the reprocessing:• Create ConfigDB entries indicating something to do • An agent submits corresponding jobs to the Grid• Grid jobs run a MAUS MC Production script, filling out details from ConfigDB entries, and store data Henry Nebrensky – MICE CM40 – 27 October 2014locally at Grid site• A Transfer Controller copies output to RAL Tape and Imperial SE (web access)Bits are there: DB stuff in Preprod DB and Launchpad; Transfer stuff in Reco; MC script?Needs putting together fast!8Miscellaneous Data Longstanding drive to archive a variety of data from other activities in MICE, e.g.:• Testbeam and cosmic data• Field Maps• Geometry and surveys• Muon Beams libraryHenry Nebrensky – MICE CM40 – 27 October 2014• EPICS Archiver archive(Data curation seems to be becoming a topic with the funding agencies)The Grid storage is my responsibility (wearing my Archivist hat) but preparation, indexing and making data available rests with its creator!https://micewww.pp.rl.ac.uk/projects/computing-software/wiki/GridDataStorage9Data Volume For data to continue to be available into the future it must be properly archived, not just left lying around.Long term storage is provided on disk and tape at various Grid sites. It doesn’t appear by magic...Henry Nebrensky – MICE CM40 – 27 October 2014... we do need realistic estimates of how much data different activities will produce, and how reliably we want it preserved. I will be chasing people soon.If data-taking has to stop because there’s nowhere to write to, that’s just plain embarrassing. Bear in mind purchase cycles look a year in advance.10FutureNear future:• Finish Batch Reprocessing run with MAUS-v0.9.1• MC Production By Step IV (i.e. January):• robot dongles for mover and reco certificatesHenry Nebrensky – MICE CM40 – 27 October 2014• (automated Data Mover at end of each run)• data distribution to RAL PPD and GlasgowThe first is mission-critical.I can only deliver deliverables if I have the time to work on them so I will be focusing on Grid for a couple of months... 11

GRID Status

http://bura.brunel.ac.uk/bitstream/2438/19830/1/FullText.pdf

GRID Status

Abstract

Similar works

Full text

Available Versions

Brunel University Research Archive