With the exponential growth of LHC (Large Hadron Collider) data in 2012, distributed computing has become the established way to analyze collider data. The ATLAS grid infrastructure includes more than 130 sites worldwide, ranging from large national computing centers to smaller university clusters. HammerCloud was previously introduced with the goals of enabling virtual organisations (VO) and site-administrators to run validation tests of the site and software infrastructure in an automated or on-demand manner. The HammerCloud infrastructure has been constantly improved to support the addition of new test workflows. These new workflows comprise e.g. tests of the ATLAS nightly build system, ATLAS Monte Carlo production system, XRootD federation (FAX) and new site stress test workflows. We report on the development, optimization and results of the various components in the HammerCloud framework

D van der Ster

F G Sciacca

F Hönig

F Legger

J Elmsheuser

null null

R Medrano LLamas

Journal of Physics Conference Series

English

Elmsheuser, J.

Hönig, F.

Legger, F.

LLamas, R. Medrano

Sciacca, F. G.

Ster, D. van der

Universität München: Elektronischen Publikationen

This content has been downloaded from IOPscience. Please scroll down to see the full text.
Download details:
IP Address: 138.246.2.253
This content was downloaded on 30/11/2016 at 16:24
Please note that terms and conditions apply.
Grid site testing for ATLAS with HammerCloud
View the table of contents for this issue, or go to the journal homepage for more
2014 J. Phys.: Conf. Ser. 513 032030
(http://iopscience.iop.org/1742-6596/513/3/032030)
Home Search Collections Journals About Contact us My IOPscience
You may also be interested in:
Improving ATLAS grid site reliability with functional tests using HammerCloud
Johannes Elmsheuser, Federica Legger, Ramon Medrano Llamas et al.
Distributed Data Analysis in the ATLAS Experiment: Challenges and Solutions
Johannes Elmsheuser and Daniel van der Ster
Experience in Grid Site Testing for ATLAS, CMS and LHCb with HammerCloud
Johannes Elmsheuser, Ramón Medrano Llamas, Federica Legger et al.
Commissioning of a CERN Production and Analysis Facility Based on xrootd
Simone Campana, Daniel C van der Ster, Alessandro Di Girolamo et al.
Testing as a Service with HammerCloud
Ramón Medrano Llamas, Quentin Barrand, Johannes Elmsheuser et al.
The ATLAS distributed analysis system
F Legger and the Atlas Collaboration
AGIS: The ATLAS Grid Information System
A Anisenkov, A Di Girolamo, A Klimentov et al.
Commissioning the CERN IT Agile Infrastructure with experiment workloads
Ramón Medrano Llamas, Fernando Harald Barreiro Megino, Katarzyna Kucharczyk et al.
Dashboard Task Monitor for Managing ATLAS User Analysis on the Grid
L Sargsyan, J Andreeva, M Jha et al.
Grid site testing for ATLAS with HammerCloud
J Elmsheuser1, F Ho¨nig1, F Legger1, R Medrano LLamas1, F G
Sciacca3 and D van der Ster2 on behalf of the ATLAS collaboration
1 Ludwig-Maximilians-Universita¨t Mu¨nchen, Germany
2 CERN, Switzerland
3 Universita¨t Bern, Switzerland
E-mail: johannes.elmsheuser@physik.uni-muenchen.de
Abstract. With the exponential growth of LHC (Large Hadron Collider) data in 2012,
distributed computing has become the established way to analyze collider data. The ATLAS grid
infrastructure includes more than 130 sites worldwide, ranging from large national computing
centers to smaller university clusters. HammerCloud was previously introduced with the goals
of enabling virtual organisations (VO) and site-administrators to run validation tests of the
site and software infrastructure in an automated or on-demand manner. The HammerCloud
infrastructure has been constantly improved to support the addition of new test workflows.
These new workflows comprise e.g. tests of the ATLAS nightly build system, ATLAS Monte
Carlo production system, XRootD federation (FAX) and new site stress test workflows.
We report on the development, optimization and results of the various components in the
HammerCloud framework.
1. Introduction
The LHC at CERN is colliding protons or heavy ions at unprecedented center-of-mass energies
and these collisions are recorded by several experiments including the ATLAS experiment [1].
The large number of collision events and resulting data volumes require an analysis on distributed
computing resources [2]. HammerCloud was previously introduced [3] as a solution to stress
the Worldwide LHC Computing Grid sites and execute functional testing on these distributed
resources.
This paper will first briefly describe the HammerCloud functionality and various test options
used within the ATLAS experiment. These tests options will be described in more detail in the
context of FAX (Federating ATLAS storage systems using XRootD) testing [4]. Some statistics
of the HammerCloud usage will be given before the conclusions.
2. HammerCloud overview
HammerCloud is a grid testing service for the ATLAS, CMS and LHCb experiments. It is
hosted by CERN IT and operated by the experiments grid operations teams. Figure 1 shows a
schematic view of the HammerCloud architecture and its various components. HammerCloud
is a Python application using GANGA [5] for test job submission and monitoring. It is using
Django as a database access layer and presents the test results to the users and operators on web
pages. GANGA is used as the backend job management tool since it provides with its plug-in
architecture a convenient access to the interfaces of the ATLAS Grid middleware stack, namely
20th International Conference on Computing in High Energy and Nuclear Physics (CHEP2013) IOP Publishing
Journal of Physics: Conference Series 513 (2014) 032030 doi:10.1088/1742-6596/513/3/032030
Content from this work may be used under the terms of the Creative Commons Attribution 3.0 licence. Any further distribution
of this work must maintain attribution to the author(s) and the title of the work, journal citation and DOI.
Published under licence by IOP Publishing Ltd 1
PanDA [6] for the workload management and DQ2 [7] for the distributed data management,
and also provides the functionality to submit, clone, resubmit and monitor a series of identical
or similar jobs.
Figure 1. Schematic view of the HammerCloud (HC) architecture and its various components.
3. HammerCloud tests
The ATLAS experiment originally developed HammerCloud as a Grid stress testing system
before the start of LHC data taking. Now, there are several extensively pursued use cases in
addition:
• Stress testing scheduled on demand: these tests are used to measure experiment applications
under different site and application configurations and evaluate their performances;
• Site functional tests: a continuous flow of jobs is sent to the PanDA analysis and production
system queues to validate their functionality. These so called “analysis functional tests”
(AFT) and “production functional tests” (PFT) are used to automatically exclude and
include sites into operations if they fail or subsequently succeed again a series of test jobs.
Email notifications are sent out to shifters, site and cloud support teams. All the procedures
have been optimized in the past months. The system works very robust and reliable;
• PanDA pilot development test jobs: the PanDA system pilot code is the middleware
component that executes the actual experiment payload job, such as the analysis or Monte
Carlo simulation code on the site worker node and interacts with all the Grid related
components. The pilot code needs to adapt to the latest Grid middleware code updates
and experiment workflow changes. This requires constant code validation which is done with
a set of continuous HammerCloud functional tests that uses the latest pilot code version
before it goes into large scale production;
• With the arrival of multiple core processors there are also special PanDA queues to use
these resources in a dedicated way. These queues are tested with a constant but low rate
of validation jobs similar to the previously described PFT. A queue exclusion/inclusion
algorithm has been developed to white-list and black-list these special queues corresponding
to the status of the regular site queues that are tested with the PFT;
• Cloud computing resources, Tier0 and higher level trigger resources are accessible through
dedicated PanDA queues. The functionality of these resources are validated with the
previously described AFT and PFT jobs;
• Athena Nightly Build System: stable releases of the ATLAS experiment oﬄine software
Athena are distributed through the CernVM file system (CVMFS) file system to all ATLAS
20th International Conference on Computing in High Energy and Nuclear Physics (CHEP2013) IOP Publishing
Journal of Physics: Conference Series 513 (2014) 032030 doi:10.1088/1742-6596/513/3/032030
2
Grid sites. Different nightly builds of this software are distributed with the same procedure
and are tested and validated with several low rate test jobs at several Grid sites. Different
Linux operating system versions and storage element types are checked for incompatibilities.
Better general overview monitoring for shifters is planned for these types of jobs;
• ROOT I/O and wide area network (WAN) tests: To ensure that changes within ROOT
versions, or changes and modifications of the analysis input file layout are tested and verified
on all different storage system technology type, test jobs are sent out at a low rate to various
different Grid sites. The performance of different features in ROOT like e.g. TTreeCache
for local and remote input file reading are tested;
• Federated XRootD (FAX): short and low rate test jobs are sent out to FAX enabled sites to
test and validate the FAX infrastructure at the site and the various global FAX components.
More details will be described in the next section.
All these various tests are defined in so called templates which define the application to be
used, the input datasets to be processed, the job submission and processing rate and the sites
to be tested. The test results about jobs successes and failures together with links to further
monitoring and debugging information is displayed on the HammerCloud web page. Figure 2
shows an example of a HammerCloud test web page.
Figure 2. Example of an individual HammerCloud test web page. Here a functional test of the
FAX system is shown.
4. HammerCloud FAX results
FAX (Federating ATLAS storage systems using XrootD) brings Tier1, Tier2 and Tier3 storage
resources together into a common namespace, accessible from anywhere, thus relaxing the
traditional requirement of data-CPU locality. Client software tools like ROOT or xrdcp interact
with FAX behind the scenes to reach data regardless of its location. Improved network
bandwidth, reduced latency and data structure aware caching mechanism like TTreeCache makes
this possible.
HammerCloud has been used to setup, test and validate the configurations of the FAX system
especially in PanDA and the PanDA pilot. Since different storage technologies and preferred
20th International Conference on Computing in High Energy and Nuclear Physics (CHEP2013) IOP Publishing
Journal of Physics: Conference Series 513 (2014) 032030 doi:10.1088/1742-6596/513/3/032030
3
input data access modes are used at every site, each site needs an individual setup for FAX
testing. In these tests the default PanDA queue configurations are overwritten on a job by job
basis using a PanDA pilot switch. These individual configurations are planned to be stored
permanently in the ATLAS Grid information system (AGIS) for easy access in the future. To
measure the performance of data throughput of the FAX system two typical ATLAS user analysis
have been used to process regular ROOT ntuples, so called D3PDs, of detector and simulation
data. Different access patterns in the FAX system have been tested; the input data access mode
using direct I/O in ROOT, and the mode using a copy of the input data to the local worker node
directory with xrdcp. These modes have been tested with access to data at the local storage
element or with remote access to the CERN storage. The FAX redirector to detect if data is
available at the local site or only remotely is setup in a hierarchical manner. There is a redirector
at the local site, in a so called cloud which combines a number of geographical close sites like
in e.g. a single country and then the global redirector. The data access and discovery using the
local and cloud redirector have been tested.
The tests have been performed in three steps. First the FAX configuration of the individual
sites has been tested with single jobs submitted from HammerCloud to verify the correct setup.
Then larger scale tests with several hundred jobs per site have been performed to test all
the different data access modes described above. Finally three different functional tests with
a low job submission rate have been setup to continuously monitor the FAX infrastructure
functionality against site glitches or PanDA pilot software changes.
Default Panda config
FAX copy-to-scratch cloud redirector
FAX copy-to-scratch local redirector
FAX copy to scratch from CERN-PRODDISK
FAX directIO local redirector
FAX directIO cloud redirector
FAX directIO local redirector from CERN-PRODDISK
Figure 3. The distribution of several HammerCloud FAX infrastructure stress tests results.
Different storage element types have been tested and event rates for several example sites are
shown. The access modes have been varied throughout the different tests and are indicated with
different markers as shown in the legend on the right.
Figure 3 shows some results of the HammerCloud infrastructure stress tests. The tests have
been performed at more than 30 FAX enabled sites. The individual tests were executed while
at the same time the regular operations of the sites continued. The test loads have been rather
moderate with 20-50 parallel running jobs and lasted 48-72 hours each. The shown event rates
are averaged over the test periods and have an error of about 10-20% each. Not all sites have
been available for all tests due to downtimes or fully occupied resources. The plot shows the
event rate for one of the described user analysis at different sites which are representative for
each of the storage element technology. Overall there is a large variation on reading speed across
the sites using the default configured PanDA input access mode, which is direct I/O for dCache,
Storm and EOS sites and copy-to-scratch on the worker node for DPM sites. For the FAX
configuration the local redirector access with direct I/O performs the best at most sites. For
DPM sites a much better performance is achieved using FAX compared to the default copy-to-
scratch mode. As expected accesses to the cloud redirector or to the remote storage at CERN
are slower.
The importance of TTreeCache usage in an user analysis is shown in figure 4 (left).
20th International Conference on Computing in High Energy and Nuclear Physics (CHEP2013) IOP Publishing
Journal of Physics: Conference Series 513 (2014) 032030 doi:10.1088/1742-6596/513/3/032030
4
CacheSize [MB]
10 210 310
E v
e n
t r a
t e
 [ H
z ]
0
200
400
600
800
1000
1200
1400
1600
1800
2000
Automatic training (185 vars)
Muons,Electrons (646 vars)
Muons,Elec,Jets,MET (2573 vars)
All (8543 vars) 
Figure 4. Left: event rate of an example analysis reading with different ROOT TTreeCache
settings. Right: Number of ATLAS jobs submitted with HammerCloud per days for the year
2013. Green indicates successful, red failed and magenta cancelled jobs.
TTreeCache acts as a file cache, registering automatically the baskets from the branches being
processed when in the learning phase. This cache speeds-up considerably the performance, in
particular when the TTree is accessed remotely via a high latency network. The user analysis
described above has been tested in a HammerCloud test using different cache sizes and number
of variables for training. The automatic training without previously defined variables shows the
best performance independent of the cache size used.
The next steps in the FAX testing using HammerCloud are large scale stress tests over longer
periods to determine the system behavior under high load. Furthermore sites using FAX will be
included into the automatic PanDA queue exclusion mechanism similar to the existing system
for analysis and production queues.
5. HammerCloud jobs
Overall HammerCloud is executing on average 50k functional test jobs per day. Periods with
more jobs per day are due to additional running stress tests. Figure 4 (right) shows the number
of ATLAS jobs submitted with HammerCloud per days for the year 2013.
6. Future plans and conclusions
HammerCloud is very successfully used in many aspects of ATLAS day-to-day Grid computing;
functional testing with site black-listing, stress testing, and new Grid component validation as
demonstrated with the FAX system setup. With the upgrade of several ATLAS Grid components
before the 2015 LHC data taking there is further integration and validation work ahead for
HammerCloud.
References
[1] ATLAS Collaboration 2008 JINST 3 S08003
[2] Duckeck D et al. [ATLAS Collaboration] 2005 “ATLAS computing: Technical design report,” ATLAS-TDR-
017; CERN-LHCC-2005-022, https://cds.cern.ch/record/837738
[3] van der Ster D C, Elmsheuser J, Ubeda Garcia M and Paladin M 2011 J. Phys. Conf. Ser. 331 072036
[4] Gardner R et al. 2013 “Data Federation Strategies for ATLAS Using XRootD” these proceedings
[5] Moscicki J T et al. 2009 “Ganga: a tool for computational-task management and easy access to Grid
resources”, Computer Physics Communications, Volume 180, Issue 11
[6] Maeno T [ATLAS Collaboration] 2008 J. Phys. Conf. Ser. 119 062036
[7] Branco M et al. [ATLAS Collaboration] 2008 J. Phys. Conf. Ser. 119 062017
20th International Conference on Computing in High Energy and Nuclear Physics (CHEP2013) IOP Publishing
Journal of Physics: Conference Series 513 (2014) 032030 doi:10.1088/1742-6596/513/3/032030
5


Grid site testing for ATLAS with Hammer Cloud

Open Access LMU

This content has been downloaded from IOPscience. Please scroll down to see the full text.Download details:IP Address: 138.246.2.253This content was downloaded on 30/11/2016 at 16:24Please note that terms and conditions apply.Grid site testing for ATLAS with HammerCloudView the table of contents for this issue, or go to the journal homepage for more2014 J. Phys.: Conf. Ser. 513 032030(http://iopscience.iop.org/1742-6596/513/3/032030)Home Search Collections Journals About Contact us My IOPscienceYou may also be interested in:Improving ATLAS grid site reliability with functional tests using HammerCloudJohannes Elmsheuser, Federica Legger, Ramon Medrano Llamas et al.Distributed Data Analysis in the ATLAS Experiment: Challenges and SolutionsJohannes Elmsheuser and Daniel van der SterExperience in Grid Site Testing for ATLAS, CMS and LHCb with HammerCloudJohannes Elmsheuser, Ramón Medrano Llamas, Federica Legger et al.Commissioning of a CERN Production and Analysis Facility Based on xrootdSimone Campana, Daniel C van der Ster, Alessandro Di Girolamo et al.Testing as a Service with HammerCloudRamón Medrano Llamas, Quentin Barrand, Johannes Elmsheuser et al.The ATLAS distributed analysis systemF Legger and the Atlas CollaborationAGIS: The ATLAS Grid Information SystemA Anisenkov, A Di Girolamo, A Klimentov et al.Commissioning the CERN IT Agile Infrastructure with experiment workloadsRamón Medrano Llamas, Fernando Harald Barreiro Megino, Katarzyna Kucharczyk et al.Dashboard Task Monitor for Managing ATLAS User Analysis on the GridL Sargsyan, J Andreeva, M Jha et al.Grid site testing for ATLAS with HammerCloudJ Elmsheuser1, F Hönig1, F Legger1, R Medrano LLamas1, F GSciacca3 and D van der Ster2 on behalf of the ATLAS collaboration1 Ludwig-Maximilians-Universität München, Germany2 CERN, Switzerland3 Universität Bern, SwitzerlandE-mail: johannes.elmsheuser@physik.uni-muenchen.deAbstract. With the exponential growth of LHC (Large Hadron Collider) data in 2012,distributed computing has become the established way to analyze collider data. The ATLAS gridinfrastructure includes more than 130 sites worldwide, ranging from large national computingcenters to smaller university clusters. HammerCloud was previously introduced with the goalsof enabling virtual organisations (VO) and site-administrators to run validation tests of thesite and software infrastructure in an automated or on-demand manner. The HammerCloudinfrastructure has been constantly improved to support the addition of new test workflows.These new workflows comprise e.g. tests of the ATLAS nightly build system, ATLAS MonteCarlo production system, XRootD federation (FAX) and new site stress test workflows.We report on the development, optimization and results of the various components in theHammerCloud framework.1. IntroductionThe LHC at CERN is colliding protons or heavy ions at unprecedented center-of-mass energiesand these collisions are recorded by several experiments including the ATLAS experiment [1].The large number of collision events and resulting data volumes require an analysis on distributedcomputing resources [2]. HammerCloud was previously introduced [3] as a solution to stressthe Worldwide LHC Computing Grid sites and execute functional testing on these distributedresources.This paper will first briefly describe the HammerCloud functionality and various test optionsused within the ATLAS experiment. These tests options will be described in more detail in thecontext of FAX (Federating ATLAS storage systems using XRootD) testing [4]. Some statisticsof the HammerCloud usage will be given before the conclusions.2. HammerCloud overviewHammerCloud is a grid testing service for the ATLAS, CMS and LHCb experiments. It ishosted by CERN IT and operated by the experiments grid operations teams. Figure 1 shows aschematic view of the HammerCloud architecture and its various components. HammerCloudis a Python application using GANGA [5] for test job submission and monitoring. It is usingDjango as a database access layer and presents the test results to the users and operators on webpages. GANGA is used as the backend job management tool since it provides with its plug-inarchitecture a convenient access to the interfaces of the ATLAS Grid middleware stack, namely20th International Conference on Computing in High Energy and Nuclear Physics (CHEP2013) IOP PublishingJournal of Physics: Conference Series 513 (2014) 032030 doi:10.1088/1742-6596/513/3/032030Content from this work may be used under the terms of the Creative Commons Attribution 3.0 licence. Any further distributionof this work must maintain attribution to the author(s) and the title of the work, journal citation and DOI.Published under licence by IOP Publishing Ltd 1PanDA [6] for the workload management and DQ2 [7] for the distributed data management,and also provides the functionality to submit, clone, resubmit and monitor a series of identicalor similar jobs.Figure 1. Schematic view of the HammerCloud (HC) architecture and its various components.3. HammerCloud testsThe ATLAS experiment originally developed HammerCloud as a Grid stress testing systembefore the start of LHC data taking. Now, there are several extensively pursued use cases inaddition:• Stress testing scheduled on demand: these tests are used to measure experiment applicationsunder different site and application configurations and evaluate their performances;• Site functional tests: a continuous flow of jobs is sent to the PanDA analysis and productionsystem queues to validate their functionality. These so called “analysis functional tests”(AFT) and “production functional tests” (PFT) are used to automatically exclude andinclude sites into operations if they fail or subsequently succeed again a series of test jobs.Email notifications are sent out to shifters, site and cloud support teams. All the procedureshave been optimized in the past months. The system works very robust and reliable;• PanDA pilot development test jobs: the PanDA system pilot code is the middlewarecomponent that executes the actual experiment payload job, such as the analysis or MonteCarlo simulation code on the site worker node and interacts with all the Grid relatedcomponents. The pilot code needs to adapt to the latest Grid middleware code updatesand experiment workflow changes. This requires constant code validation which is done witha set of continuous HammerCloud functional tests that uses the latest pilot code versionbefore it goes into large scale production;• With the arrival of multiple core processors there are also special PanDA queues to usethese resources in a dedicated way. These queues are tested with a constant but low rateof validation jobs similar to the previously described PFT. A queue exclusion/inclusionalgorithm has been developed to white-list and black-list these special queues correspondingto the status of the regular site queues that are tested with the PFT;• Cloud computing resources, Tier0 and higher level trigger resources are accessible throughdedicated PanDA queues. The functionality of these resources are validated with thepreviously described AFT and PFT jobs;• Athena Nightly Build System: stable releases of the ATLAS experiment offline softwareAthena are distributed through the CernVM file system (CVMFS) file system to all ATLAS20th International Conference on Computing in High Energy and Nuclear Physics (CHEP2013) IOP PublishingJournal of Physics: Conference Series 513 (2014) 032030 doi:10.1088/1742-6596/513/3/0320302Grid sites. Different nightly builds of this software are distributed with the same procedureand are tested and validated with several low rate test jobs at several Grid sites. DifferentLinux operating system versions and storage element types are checked for incompatibilities.Better general overview monitoring for shifters is planned for these types of jobs;• ROOT I/O and wide area network (WAN) tests: To ensure that changes within ROOTversions, or changes and modifications of the analysis input file layout are tested and verifiedon all different storage system technology type, test jobs are sent out at a low rate to variousdifferent Grid sites. The performance of different features in ROOT like e.g. TTreeCachefor local and remote input file reading are tested;• Federated XRootD (FAX): short and low rate test jobs are sent out to FAX enabled sites totest and validate the FAX infrastructure at the site and the various global FAX components.More details will be described in the next section.All these various tests are defined in so called templates which define the application to beused, the input datasets to be processed, the job submission and processing rate and the sitesto be tested. The test results about jobs successes and failures together with links to furthermonitoring and debugging information is displayed on the HammerCloud web page. Figure 2shows an example of a HammerCloud test web page.Figure 2. Example of an individual HammerCloud test web page. Here a functional test of theFAX system is shown.4. HammerCloud FAX resultsFAX (Federating ATLAS storage systems using XrootD) brings Tier1, Tier2 and Tier3 storageresources together into a common namespace, accessible from anywhere, thus relaxing thetraditional requirement of data-CPU locality. Client software tools like ROOT or xrdcp interactwith FAX behind the scenes to reach data regardless of its location. Improved networkbandwidth, reduced latency and data structure aware caching mechanism like TTreeCache makesthis possible.HammerCloud has been used to setup, test and validate the configurations of the FAX systemespecially in PanDA and the PanDA pilot. Since different storage technologies and preferred20th International Conference on Computing in High Energy and Nuclear Physics (CHEP2013) IOP PublishingJournal of Physics: Conference Series 513 (2014) 032030 doi:10.1088/1742-6596/513/3/0320303input data access modes are used at every site, each site needs an individual setup for FAXtesting. In these tests the default PanDA queue configurations are overwritten on a job by jobbasis using a PanDA pilot switch. These individual configurations are planned to be storedpermanently in the ATLAS Grid information system (AGIS) for easy access in the future. Tomeasure the performance of data throughput of the FAX system two typical ATLAS user analysishave been used to process regular ROOT ntuples, so called D3PDs, of detector and simulationdata. Different access patterns in the FAX system have been tested; the input data access modeusing direct I/O in ROOT, and the mode using a copy of the input data to the local worker nodedirectory with xrdcp. These modes have been tested with access to data at the local storageelement or with remote access to the CERN storage. The FAX redirector to detect if data isavailable at the local site or only remotely is setup in a hierarchical manner. There is a redirectorat the local site, in a so called cloud which combines a number of geographical close sites likein e.g. a single country and then the global redirector. The data access and discovery using thelocal and cloud redirector have been tested.The tests have been performed in three steps. First the FAX configuration of the individualsites has been tested with single jobs submitted from HammerCloud to verify the correct setup.Then larger scale tests with several hundred jobs per site have been performed to test allthe different data access modes described above. Finally three different functional tests witha low job submission rate have been setup to continuously monitor the FAX infrastructurefunctionality against site glitches or PanDA pilot software changes.Default Panda configFAX copy-to-scratch cloud redirectorFAX copy-to-scratch local redirectorFAX copy to scratch from CERN-PRODDISKFAX directIO local redirectorFAX directIO cloud redirectorFAX directIO local redirector from CERN-PRODDISKFigure 3. The distribution of several HammerCloud FAX infrastructure stress tests results.Different storage element types have been tested and event rates for several example sites areshown. The access modes have been varied throughout the different tests and are indicated withdifferent markers as shown in the legend on the right.Figure 3 shows some results of the HammerCloud infrastructure stress tests. The tests havebeen performed at more than 30 FAX enabled sites. The individual tests were executed whileat the same time the regular operations of the sites continued. The test loads have been rathermoderate with 20-50 parallel running jobs and lasted 48-72 hours each. The shown event ratesare averaged over the test periods and have an error of about 10-20% each. Not all sites havebeen available for all tests due to downtimes or fully occupied resources. The plot shows theevent rate for one of the described user analysis at different sites which are representative foreach of the storage element technology. Overall there is a large variation on reading speed acrossthe sites using the default configured PanDA input access mode, which is direct I/O for dCache,Storm and EOS sites and copy-to-scratch on the worker node for DPM sites. For the FAXconfiguration the local redirector access with direct I/O performs the best at most sites. ForDPM sites a much better performance is achieved using FAX compared to the default copy-to-scratch mode. As expected accesses to the cloud redirector or to the remote storage at CERNare slower.The importance of TTreeCache usage in an user analysis is shown in figure 4 (left).20th International Conference on Computing in High Energy and Nuclear Physics (CHEP2013) IOP PublishingJournal of Physics: Conference Series 513 (2014) 032030 doi:10.1088/1742-6596/513/3/0320304CacheSize [MB]10 210 310Eventrate [Hz]0200400600800100012001400160018002000Automatic training (185 vars)Muons,Electrons (646 vars)Muons,Elec,Jets,MET (2573 vars)All (8543 vars) Figure 4. Left: event rate of an example analysis reading with different ROOT TTreeCachesettings. Right: Number of ATLAS jobs submitted with HammerCloud per days for the year2013. Green indicates successful, red failed and magenta cancelled jobs.TTreeCache acts as a file cache, registering automatically the baskets from the branches beingprocessed when in the learning phase. This cache speeds-up considerably the performance, inparticular when the TTree is accessed remotely via a high latency network. The user analysisdescribed above has been tested in a HammerCloud test using different cache sizes and numberof variables for training. The automatic training without previously defined variables shows thebest performance independent of the cache size used.The next steps in the FAX testing using HammerCloud are large scale stress tests over longerperiods to determine the system behavior under high load. Furthermore sites using FAX will beincluded into the automatic PanDA queue exclusion mechanism similar to the existing systemfor analysis and production queues.5. HammerCloud jobsOverall HammerCloud is executing on average 50k functional test jobs per day. Periods withmore jobs per day are due to additional running stress tests. Figure 4 (right) shows the numberof ATLAS jobs submitted with HammerCloud per days for the year 2013.6. Future plans and conclusionsHammerCloud is very successfully used in many aspects of ATLAS day-to-day Grid computing;functional testing with site black-listing, stress testing, and new Grid component validation asdemonstrated with the FAX system setup. With the upgrade of several ATLAS Grid componentsbefore the 2015 LHC data taking there is further integration and validation work ahead forHammerCloud.References[1] ATLAS Collaboration 2008 JINST 3 S08003[2] Duckeck D et al. [ATLAS Collaboration] 2005 “ATLAS computing: Technical design report,” ATLAS-TDR-017; CERN-LHCC-2005-022, https://cds.cern.ch/record/837738[3] van der Ster D C, Elmsheuser J, Ubeda Garcia M and Paladin M 2011 J. Phys. Conf. Ser. 331 072036[4] Gardner R et al. 2013 “Data Federation Strategies for ATLAS Using XRootD” these proceedings[5] Moscicki J T et al. 2009 “Ganga: a tool for computational-task management and easy access to Gridresources”, Computer Physics Communications, Volume 180, Issue 11[6] Maeno T [ATLAS Collaboration] 2008 J. Phys. Conf. Ser. 119 062036[7] Branco M et al. [ATLAS Collaboration] 2008 J. Phys. Conf. Ser. 119 06201720th International Conference on Computing in High Energy and Nuclear Physics (CHEP2013) IOP PublishingJournal of Physics: Conference Series 513 (2014) 032030 doi:10.1088/1742-6596/513/3/0320305

Crossref

Grid site testing for ATLAS with HammerCloud

https://epub.ub.uni-muenchen.de/33831/1/10.1088_1742-6596_513_3_032030.pdf

Grid site testing for ATLAS with Hammer Cloud

Abstract

Similar works

Full text

Available Versions

Universität München: Elektronischen Publikationen

Open Access LMU

Crossref