Presentation at 39th MICE Collaboration Meeting (CM39), 25th-28th June 2014. https://indico.cern.ch/event/321200/timetable/ (Slides titled "Grid Status", updated 27th June 2014

Nebrensky, J

English

Brunel University Research Archive

Grid StatusHenry NebrenskyBrunel UniversityHenry Nebrensky – MICE CM39 updated – 27 June 2014 1Data Mover Permanent storage of data from the MICE DAQ.• Initial transfer/“File Compactor” still triggered manually (moveFiles script)• Autonomous process - “Data Mover” – then makes copy at RAL Tier 1 for tape archival.Henry Nebrensky – MICE CM39 – 26 June 2014• Separate agents then make copies at Imperial and BrunelMLCR DataMover PC is being moved to SL6, will move to state tracking in MySQL DB, other tweaks. 2MICE Raw Data Raw data on Grid, for up to October 2013 run inclusive • Data validation up to last October (run 5683)• All data tarballs valid (internal checksums OK)• All data tarballs have two copies on tape at Tier 1 CastorHenry Nebrensky – MICE CM39 – 26 June 2014• All data tarballs have overall checksums match between Castor and MLCR copy• Copies of data at RAL PPD, Imperial College (web) and Brunel (Grid/WebDAV)3MICE Data Copies of data at RAL PPD, Imperial College (web) and Brunel (Grid/WebDAV)Henry Nebrensky – MICE CM35 – 15 February 2013 4CVMFSCVMFS • a read-only filesystem based on HTTP; uses caching to give (usable) global coverage• the master copy, Stratum-0, is at the RAL Tier1• installed on Grid clusters at Brunel, Imperial and RAL PPD and Tier 1Henry Nebrensky – MICE CM39 – 26 June 2014• MAUS is compiled and built at Imperial, then the binaries moved to the Stratum-0 and replicated across the Grid• Now a manual, interactive process so we get more feedback during the install• Have so far installed MAUS 0.6.0, 0.8.1, 0.8.55Offline Reco MAUS installed at RAL Tier1 for Offline Reco • automated process waiting for new raw data files• when a new data file appears, makes Offline Reco and run-specific Monte-Carlo job using latest approved MAUS • uses fast-response queue at RAL Tier1Henry Nebrensky – MICE CM39 – 26 June 2014• automatically runs MAUS jobs, creates output tarballs, saves them to Grid disk and tape :)Steps tested in principle during EMR run. Needs automation (dongle) and exercising of the system. 6Batch Reprocessing A manually-triggered process, in response to a particular MAUS release becoming available via CVMFS:• makes Offline Reco and run-specific Monte-Carlo job using that MAUS release for every run in a specified Step. Henry Nebrensky – MICE CM39 – 26 June 2014• uses Tier2 sites across the UK• automatically runs MAUS jobs, creates output tarballs, saves them to Grid disk and tape :)• Demonstrated last autumn with MAUS-v0.6.0 –took a few weeksMAUS 0.8.1 and 0.8.5 failing with major memory leaks (killed at 6 GB)7Reco Data• Saved to Grid disk and tape, with web accessHenry Nebrensky – MICE CM39 – 26 June 2014 8Miscellaneous Data Longstanding drive to archive a variety of data from other activities in MICE, e.g.:• Testbeam and cosmic data• Field Maps• Geometry and surveys• Muon Beams libraryHenry Nebrensky – MICE CM39 – 26 June 2014• EPICS Archiver archive(Data curation seems to be becoming a topic with the funding agencies)The Grid storage is my responsibility (wearing my Archivist hat) but preparation, indexing and making data available rests with its creator!https://micewww.pp.rl.ac.uk/projects/computing-software/wiki/GridDataStorage9Other Data• Saved to Grid disk and tape, with web accessHenry Nebrensky – MICE CM39 – 26 June 2014 10FutureNear future:• Batch Offline Reco run with MAUS-v0.8x• Data Mover to SL6Step IV:• automated Data Mover at end of each runHenry Nebrensky – MICE CM39  updated – 27 June 2014• robot dongles for mover and reco certificates• data distribution to RAL PPD and Glasgow?:• MC Production 11

GRID

http://bura.brunel.ac.uk/bitstream/2438/19901/1/FullText%20%281%29.pdf

GRID

Abstract

Similar works

Full text

Available Versions

Brunel University Research Archive