The Linux kernel feature Control Groups (cgroups) has been used to gather metrics on the resource usage of single and eight-core ATLAS workloads. It has been used to study the effects on performance of a reduction in the amount of physical memory. The results were used to optimise cluster performance, and consequently increase cluster throughput by up to 10%

Britton, D.

Crooks, D.

Qin, G.

Roy, G.

Skipsey, S.C.

Stewart, G.P.

Journal of Physics : Conference Series

English

G Qin

G Roy

D Crooks

S C Skipsey

G P Stewart

D Britton

Crossref

Journal of Physics Conference Series

Cluster Optimisation using Cgroups at a Tier-2

Enlighten

Journal of Physics: Conference SeriesPAPER • OPEN ACCESSCluster Optimisation using Cgroups at a Tier-2To cite this article: G Qin et al 2016 J. Phys.: Conf. Ser. 762 012010 View the article online for updates and enhancements.Related contentOptimisation of 1.3-\mu m InAs/GaAsQuantum-Dot Lasers Monolithically Grownon Si SubstratesMingchu Tang, Siming Chen, Jiang Wu etal.-Optimisation of the ATLAS TrackReconstruction Software for Run-2Andreas Salzburger-Optimisation of two-stage screwexpanders for waste heat recoveryapplicationsM G Read, I K Smith and N Stosic-This content was downloaded from IP address 130.209.115.202 on 16/10/2017 at 08:52Cluster Optimisation using Cgroups at a Tier-2G Qin, G Roy, D Crooks, S C Skipsey, G P Stewart and D BrittonSchool of Physics and Astronomy, Kelvin Building, University of Glasgow, GLASGOW, G128QQ, United KingdomE-mail: gang.qin@glasgow.ac.uk, gareth.roy@glasgow.ac.ukAbstract. The Linux kernel feature Control Groups (cgroups) has been used to gather metricson the resource usage of single and eight-core ATLAS workloads. It has been used to study theeffects on performance of a reduction in the amount of physical memory. The results were usedto optimise cluster performance, and consequently increase cluster throughput by up to 10%.1. IntroductionBeginning in 2015, the ATLAS experiment started to deliver eight-core production workloadsto the sites of the WLCG. The purpose of moving to eight-core production was to significantlyreduce the overall memory usage of production payloads, and to remain within the ‘1-CPU to2 GB of physical RAM’ regime. After the introduction of eight-core payloads, UKI-SCOTGRID-GLASGOW saw an overall reduction in site utilisation, caused by CPU cores being unused whenthe ideal mix of single and eight-core jobs was not available. In an attempt to maximise sitethroughput, a study was undertaken comparing the memory allocation requested by ATLASeight-core payloads to the amount of memory actually used by such jobs.Accurately measuring resource usage required a technology that would not double-countshared memory, which had been previously identified as an issue by ATLAS[1]. Control Groups(cgroups) [2] is a Linux kernel feature that supports resource allocation among user-definedgroups or processes. Importantly, cgroups also accurately reports the memory usage of runningprocesses in a variety of different ways, which allows accurate usage numbers to be extracted.To gain a deeper understanding of cgroups and to better understand the performance ofATLAS eight-core simulation payloads, a study was carried out which capped the availablephysical memory of jobs. This allowed the performance and behaviour of such jobs in memory-starved environments to be examined.Subsequently, cgroups were enabled on the entire production HTCondor (hereafter referredto as ‘Condor’) cluster in late 2014. Using the existing cgroups hierarchy put in place by Condor,a system was created to collect and aggregate metrics extracted from all running batch jobs.From data collected between January and December 2015, memory and CPU usage footprintswere extracted and the type of ATLAS[3] workload was identified. This data has been usedto optimise the amount of memory allocated to ATLAS workloads (initially ATLAS single-coreproduction workloads) in order to increase cluster utilisation by up to 10%.ACAT2016 IOP PublishingJournal of Physics: Conference Series 762 (2016) 012010 doi:10.1088/1742-6596/762/1/012010Content from this work may be used under the terms of the Creative Commons Attribution 3.0 licence. Any further distributionof this work must maintain attribution to the author(s) and the title of the work, journal citation and DOI.Published under licence by IOP Publishing Ltd 12. Performance in a Memory-constrained EnvironmentA series of tests were carried out to study how ATLAS eight-core simulation workloads couldbe affected by constraining the amount of available physical memory. Tests were carried out ona dual Intel Xeon X5650 system with 12 physical cores (or 24 Hyper-Threaded cores), 24 GB ofDDR3 SDRAM and 24 GB of available swap space on disk. The test platform was configured as aCondor worker node with a cgroup hard limit which was varied between 2600 MB and 1000 MB insteps of 200 MB. Initial results were obtained by running a single ATLAS AthenaMP eight-coresimulation workload which would have requested 8 CPU cores and 16 GB of physical memory.For each workload which was run, the resident set size (RSS) memory usage, swap space usage(SWAP), total memory usage (MEMORY, equal to RSS + SWAP) and CPU usage (CPU) wasextracted for the lifetime of the job. In this case, the RSS is the actual physical memory usageand similar in concept to Proportional Set Size (PSS)[4] used in batch systems like SLURM.Time (Hour)0 0.5 1 1.5 2 2.5RSS (MB)05001000150020002500300035004000(a) 10000MB 2600MB 2400MB2200MB2000MB1800MB1600MB1400MB1200MB1000MBTime (Hour)0 0.5 1 1.5 2 2.5SWAP (MB)05001000150020002500300035004000(b) 10000MB 2600MB 2400MB2200MB2000MB1800MB1600MB1400MB1200MB1000MBTime (Hour)0 0.5 1 1.5 2 2.5MEMORY (MB)0500100015002000250030003500400045005000(c) 10000MB 2600MB 2400MB2200MB2000MB1800MB1600MB1400MB1200MB1000MBTime  (Hour)0 0.5 1 1.5 2 2.5CPU 024681012(d) 10000MB 2600MB 2400MB2200MB2000MB1800MB1600MB1400MB1200MB1000MBFigure 1. (a) RSS footprints of a single 8-core ATLAS simulation workload; (b) SWAPfootprints of these workload; (c) MEMORY footprints of these workload; (d) CPU footprints ofthese wokload.Figure 1 illustrates the results obtained from this study. From Figure 1(c) it can be observedthat the peak memory usage was approximately 3500 MB even though the requested memoryallocation would have been 16 GB, which leads to a 4.5 times under-utilisation of memory. FromFigure 1(a) it can be seen that the overall running time of the workload is not greatly impacteduntil the memory limit is reduced below 1400MB, and Figure 1(d) shows that below this the CPUutilisation is affected for the whole of the job’s lifetime due to the need to constantly stage datafrom the swap partition. These results demonstrate that, for this particular payload, a 16 GBrequirement for memory usage under-utilises system resources, and that moderate amount ofswapping does not necessarily affect the running time of the simulation.ACAT2016 IOP PublishingJournal of Physics: Conference Series 762 (2016) 012010 doi:10.1088/1742-6596/762/1/01201023. Resource Usage of ATLAS JobsHaving examined the performance of typical ATLAS workloads on an isolated node, the nextstep involved investigating how adjustments to the cgroups configuration would affect theperformance of our cluster as a whole. This entailed the development of tools to extractvarious pieces of useful information from both the local cluster and from remote ATLAS servers.Parameters for approximately 2.5 million jobs such as MaximumMemory, MaximumCpuTime,AverageMemory, and Runtime were collect during 2015.3.1. ATLAS Eight-core SimulationDuring 2015, details of about 55,000 successfully-completed ATLAS eight-core simulation jobswere collected. Figure 2(a) shows the distribution of MaximumMemory with a maximum valueof 11 GB and a mean value of 5.66 GB, along with the distribution of AverageMemory which hasa maximum value of 10 GB and a mean of 4.81 GB. The overall AverageCpu usage is illustratedin Figure 2(b), with the corresponding normalised efficiency being 94%. Figure 2(c) shows thatthe mean Runtime of these jobs is 4.3 hours. Runtimes vary up to 48 hours, but only 1.65% runfor longer than 24 hours.Size (GB)0 2 4 6 8 10 12 14Number of Jobs0500100015002000250030003500400045005000 MaximumMemory−−−−−−  AverageMemory−−−−−−(a)Cpu 0 1 2 3 4 5 6 7 8 9Number of Jobs0200040006000800010000(b)Time (Hour)0 5 10 15 20Number of Jobs010002000300040005000 (c)Figure 2. (a) Distribution of MaximumMemory & AverageMemory; (b) Distribution ofAverageCpu; (c) Distribution of Runtime.3.2. ATLAS Eight-core Reconstruction JobIn the same period, details of about 29,000 successfully-completed ATLAS eight-corereconstruction jobs were collected. Figure 3(a) shows that the mean MaximumMemory valueis about 11 GB with a maximum of 19 GB, while AverageMemory shows a mean of 6 GBand a maximum of 14 GB. The overall AverageCpu usage is illustrated in Figure 3(b) with acalculated normalised CPU efficiency of 61%. Figure 3(c) shows the Runtime of these jobs, witha mean value of ∼2.8 hours. In a very small proportion of jobs, a maximum Runtime of up to48 hours was observed, but only 0.3% of jobs run for longer than 24 hours, with a peak Runtimecentred at ∼1.5 hours. At UKI-SCOTGRID-GLASGOW, all production queues have a 48-hourACAT2016 IOP PublishingJournal of Physics: Conference Series 762 (2016) 012010 doi:10.1088/1742-6596/762/1/0120103maximum Runtime; given the results from Figure 2 and Figure 3, this could be easily reducedto 24 hours, or even much lower, to allow better scheduling of resources.Size (GB)0 2 4 6 8 10 12 14 16 18 20 22 24Number of Jobs05001000150020002500 MaximumMemory−−−−−−  AverageMemory−−−−−−(a)Cpu 0 1 2 3 4 5 6 7 8 9Number of Jobs0100200300400500600700800(b)Time (Hour)0 2 4 6 8 10Number of Jobs05001000150020002500(c)Figure 3. (a) Distribution of MaximumMemory & AverageMemory; (b) Distribution ofAverageCpu; (c) Distribution of Runtime.3.3. ATLAS Single-core ProductionDuring 2015, details of about 294,000 successfully-completed ATLAS single-core production jobswere collected. Figure 4(a) shows that the mean MaximumMemory value is ∼0.98 GB with amaximum of ∼2.4 GB, while AverageMemory shows a mean of 0.83 GB and a maximum of∼2.3 GB. AverageCpu usage has a mean value of 0.96, as illustrated in Figure 4(b). Figure 4(c)shows the Runtime of these jobs, with a mean value of ∼5.63 hours; about half finished withintwo hours, while 3.5% have a Runtime in excess of one day.3.4. ATLAS Single-core AnalysisDuring 2015, details of about 463,000 successfully-completed ATLAS analysis single-core jobswere collected. Figure 5(a) shows that the mean MaximumMemory value is ∼0.41 GB witha maximum of ∼1.5 GB, while AverageMemory shows a mean of 0.32 GB and a maximum of∼1.5 GB. The overall AverageCpu usage is 0.79, as illustrated in Figure 5(b). Figure 5(c) showsthat approximately 90% of jobs finished within five hours, and only 1.7% of jobs ran for longerthan one day.4. Evaluation of Memory UtilisationTable 1 summarises the results obtained from the above studies. From Table 1 it can be seenthat the amount of memory requested for ATLAS 8-core reconstruction workloads correctlyestimates their requirement. However, for all other ATLAS workloads, the requested memoryover-estimates the requirement significantly, and in the case of single-core analysis does so by afactor of two. In the case of analysis, this over-estimation is a function of the payloads executedat UKI-SCOTGRID-GLASGOW and may not be indicative of all other sites.ACAT2016 IOP PublishingJournal of Physics: Conference Series 762 (2016) 012010 doi:10.1088/1742-6596/762/1/0120104Size (GB)-0.5 0 0.5 1 1.5 2 2.5 3 3.5 4Number of Jobs050001000015000200002500030000350004000045000 MaximumMemory−−−−−−  AverageMemory−−−−−−(a)Cpu 0 0.2 0.4 0.6 0.8 1 1.2Number of Jobs010000200003000040000500006000070000(b)Time (Hour)0 5 10 15 20Number of Jobs050001000015000200002500030000350004000045000 (c)Figure 4. (a) Distribution of MaximumMemory & AverageMemory; (b) Distribution ofAverageCpu; (c) Distribution of Runtime.Size (GB)-0.5 0 0.5 1 1.5 2 2.5 3 3.5 4Number of Jobs020406080100120140160180200310× MaximumMemory−−−−−−  AverageMemory−−−−−−(a)Cpu 0 0.2 0.4 0.6 0.8 1 1.2Number of Jobs020406080100120310×(b)Time (Hour)0 5 10 15 20Number of Jobs010000200003000040000500006000070000 (c)Figure 5. (a) Distribution of MaximumMemory and AverageMemory; (b) Distribution ofAverageCpu; (c) Distribution of Runtime.There are two changes that ATLAS could make to improve utilisation of system resources.Firstly, if they were to distinguish between eight-core simulation and eight-core reconstructionjobs at the queue or pilot level, an improvement in utilisation of up to 31% could be obtained.Secondly, if the memory requested for single-core jobs was reduced to ∼2GB, a significantincrease in system utilisation could be achieved.ACAT2016 IOP PublishingJournal of Physics: Conference Series 762 (2016) 012010 doi:10.1088/1742-6596/762/1/0120105Table 1. Memory requested by various types of ATLAS job compared to memory actually used.Job Description CoresRequestedMemoryRequestedMaximumMemoryUsedAverageMemoryUsed1 8-core Simulation 8 16 GB 11.0 GB 5.66 GB2 8-core Reconstruction 8 16 GB 19.0 GB 11.00 GB3 1-core Production 1 3 GB 2.4 GB 0.98 GB4 1-core Analysis 1 4 GB 1.5 GB 0.41 GB5. Optimisation of Cluster ResourcesAs discussed in the introduction, with the inclusion of eight-core payloads an overall reductionin site utilisation occurred when the correct mix of single-core to eight-core payloads wasunavailable. This was due to older generations of hardware not being easily quantised intounits of eight cores. Assuming the data presented on memory utilisation was indicative of allexisting ATLAS workloads, it was possible to better utilise local resources by modifying theamount of memory allocated to each job. At first, reducing the memory allocation for single-core jobs to 1 GB of physical memory (based on average usage) allowed all unused cores to beused, resulting in a significant increase in overall cluster utilisation. However, this reduction inmemory resulted in jobs being killed due to worker nodes running out of memory and excessivelyswapping. In order to better utilise resources without causing failed jobs, a value of 1.8 GB persingle-core job was selected (Table 2 illustrates suggestions for additional limits for other ATLASworkloads). By doing this, it was observed that the number of used cores increased to ∼460 withonly ∼180 unused cores, representing an improvement in cluster utilisation of ∼10%. Clusteroperation in this configuration was observed over a period of six months, and it was found thatthis setting provided the greatest improvement in utilisation without risking instability due tothe overloading of machines.Table 2. Suggested memory allocations for various types of ATLAS job, compared to requestedmemory.Job Description CoresRequestedMemoryRequestedMemoryAllocatedSaving1 8-core Simulation 8 16 GB 12.0 GB 25%2 8-core Reconstruction 8 16 GB 16.0 GB 0%3 1-core Production 1 3 GB 1.8 GB 40%4 1-core Analysis 1 4 GB 1.5 GB 63%6. ConclusionsIn this paper we have introduced a system used to extract memory and CPU footprints fromthe UKI-SCOTGRID-GLASGOW HTCondor cluster. We have examined the detailed resourceusage of single and eight-core ATLAS workloads. Based on the results, the site has adjusted itsscheduling policy with the result that an increase in cluster utilisation of approximately 10%has been obtained.ACAT2016 IOP PublishingJournal of Physics: Conference Series 762 (2016) 012010 doi:10.1088/1742-6596/762/1/0120106References[1] Status and Future Evolution of the ATLAS Oﬄine Software, R. Seuster et al., J. Phys.: Conf. Ser., 664(7)072044, 2015.[2] Red Hat Enterprise Linux 6 Resource Management Guide, P. Ondrejka et al., 2013.[3] ATLAS Computing Technical Design Report, 2005.[4] Formulating Optimized Storage and Memory Space Specifications for Linux Network Embedded Systems, K.Tsiligkos and A. Meliones, HPCC/CSS/ICESS, pp. 580–584., IEEE, 2014.ACAT2016 IOP PublishingJournal of Physics: Conference Series 762 (2016) 012010 doi:10.1088/1742-6596/762/1/0120107

Cluster optimisation using Cgroups at a tier-2

Enlighten: Publications

http://eprints.gla.ac.uk/149786/7/149786.pdf

Cluster optimisation using Cgroups at a tier-2

Abstract

Similar works

Full text

Available Versions

Crossref

Enlighten

Enlighten: Publications