Caspart, René

Fischer, Max

Giffels, Manuel

Kühn, Eileen

Quast, Günter

Schnepf, Matthias J.

von Cube, R. Florian

English

Cube, R. Florian von

KITopen

Dynamic Computing Resource Extensionusing COBalD/TARDISR. Florian von Cube, René Caspart, Max Fischer, Manuel Giels, Eileen Kühn,Günter Quast, and Matthias J. SchnepfKarlsruhe Institute of Technology, Germany{florian.cube, rene.caspart, max.fischer, manuel.giffels, eileen.kuehn,guenter.quast, matthias.schnepf}@kit.eduAbstract. To dynamically increase computing power on demand, Cloudproviders, HPC clusters, and free institute resources can be used. In orderto make theses so-called opportunistic resources transparently available,the services COBalD and TARDIS are developed in collaboration ofthe Institute of Experimental Particle Physics (ETP) and the SteinbuchCentre for Computing (SCC) at KIT. The opportunistic resources areintegrated into an overlay batch system (OBS), which acts as a single-point-of-entry for the users. Depending on the decisions of the OBS, theutilization, and the allocation of the additional compute resources, CO-BalD/TARDIS adjust the resource allocation at the various resourceproviders. To supply the necessary software environment for the jobs, re-quired by the scientic communities, virtualization and containerizationtechnologies are used on those heterogeneous resources.Keywords: Opportunistic Resources · Heterogeneous Resources · Over-lay Batch System · High Energy Physics · Resource Scheduling.1 Computing Challenges in PhysicsWith future experiments, recording drastically increasing amounts of data,more computing resources will be needed for analysis. In those analyses, physi-cists want to exploit modern techniques which require specic hardware, as e.g.graphic cards. Many institutes in physics have dedicated local worker nodeswhich are made accessible for the end-users through a batch system. Those stat-ically integrated resources are adapted to the physicists needs, providing thenecessary soft- and hardware, but are usually provisioned for the average re-source demands. With uctuating demand in computing power, this can resultin long waiting times during peak loads.As the landscape in which scientic computing takes place shifts from com-munity-specific dedicated computing clusters, to shared science computing cen-ters, HPC clusters, and cloud providers, already today, those can be used forCopyright © 2020 for this paper by its authors. Use permitted under CreativeCommons License Attribution 4.0 International (CC BY 4.0).mitigating load peaks or providing specic hardware. By backlling with jobsfrom other communities, those resources can achieve higher eciencies. The us-age of the shared clusters, however, is often inconvenient for the end-user andrequires a lot of eort, because of multiple identity providers, the need to pro-vision software and each cluster operating its dedicated workload managementsystem. With several such resources, the end-user has to assess which the mostsuitable resource is and check which resource has capacity at the current time. Ifa resource is only temporarily available, this decision gets even more challenging.2 Single point-of-entryFor the end-user, usage through a single point-of-entry with dynamic integrationand provisioning of a heterogeneous resource-pool, containing all resources, alsothe opportunistic ones, is more desirable. To enable a transparent usage, theopportunistic resources are integrated into an overlay batch-system (OBS). Thiscan be e.g. the batch system of the institute or the collaboration. By integrat-ing the opportunistic resources into the OBS, the end-user only has to interactwith this layer, instead of with each resource separately. Authentication and au-thorization at the several opportunistic resources is done through proxy usersper institute or research group. With this further authentication of the end userbecomes dispensable.The OBS also takes care of the decision of the job to resource schedulingand accounting. However, providing the user with a single point-of-entry to aplethora of heterogeneous resources, it has to be taken care, that the jobs run-time environment is the same, no matter which resource it is scheduled to. Forthis, containerization and virtualization technologies are used. With those, it isassured, that e.g. specic software is installed.3 Resource Integration using TARDISThe integration of the opportunistic resources is performed by TARDIS [11](Transparent Adaptive Resource Dynamic Integration System). TARDIS startsso called drones on the opportunistic resource, which integrate the resource intothe OBS. A drone is a placeholder job that allocates the resource and integratesit in the OBS. The drone runs either natively on the resource, or it is startedin a container, or a virtual machine. The drone is responsible for preparing theresource to accept jobs from the OBS. When this preparation is performed, theOBS can place jobs on the newly integrated resource.3.1 Overlay Batch-System AdapterFor the integration of the opportunistic resources into the OBS, TARDIS in-terfaces the OBS through a BatchSystemAdapter. The adapter denes, howresources are added to the OBS and how they are managed. At the momentTARDIS supports HTCondor [12] as OBS. An adapter for the Slurm Work-load Manager is currently under development in collaboration with the universityof Freiburg.3.2 Resource Provider InterfaceFor the life cycle management of the opportunistic resources, they are interfacedwith SiteAdapters. A SiteAdapter denes, how the resource is allocated, howthe status of the drone can be monitored, and how drones can be terminated.At the moment HTCondor, MOAB [6], and Slurm [15] batch systems, aswell as CloudStack [7], and OpenStack [14] cloud APIs are supported.4 Resource Mix Assessment with COBalDTo optimize the usage of the integrated opportunistic resources, the decisions ofthe OBS are monitored by assessing the suitability of the presently integratedresources to the current job mix. This assessment is done by COBalD [9] (CO-BalD  the Opportunistic Balancing Daemon).For each resource component like CPU, memory, or disk space, the ratioof the assigned and the requested component is determined. Here, requestedmeans requested by the drone, and assigned is what is assigned to the actualjobs running on the resource. The maximum and minimum are called allocationand utilization, respectively:allocation = max(CPUassignedCPUrequested,MemoryassignedMemoryrequested, · · ·)utilization = min(CPUassignedCPUrequested,MemoryassignedMemoryrequested, · · ·).By comparing those, a suitability of the drone to the current job mix canbe determined. Consider the following example as shown in g. 1: A dronerequesting a virtual machine with 20 CPU-cores and 80 GB of memory at theresource provider, runinng jobs which sum up to 20 CPU cores, but only use60 GB of memory. With this the machine is fully allocated as all CPU-cores areassigned and it can not run any more jobs. However the memory is only utilizedto 75%. This implies that the machine is not the optimal t for the current jobmix.Depending on the utilization and the allocation of the drones, COBalD de-termines the demand for the respective resource. That is the number of dronesof that type that should be requested. On the other hand supply is the numberof drones of the same type, currently available in the OBS. The policy, whento request more drones of a specic type is congurable with a Controller. Itincreases the demand, when drones of the specic type have a high allocationand decreases it if the utilization is low.notusedutilizationallocationCPUFractionMemory Fraction11Fig. 1. Depiction of the metrics allocation and utilization used by COBalD toassess the suitability of the drone to the presently running jobs. The jobs are shownas the darker rectangles. The gray area is a resource component fraction, not used byany job. The values measured are directly inuenced by the decisions of the overlaybatch system and thus can be used for reacting accordingly.COBalD uses TARDIS resource provider interfaces to determine the num-ber of drones which should be requested at each one of them. An overview of theresource scheduling process, depending on the decisions of the OBS is shown ing. 2. Because of their modularity COBalD/TARDIS are easily customizableand can be adapted to virtually every setup.5 Examplary SetupsCOBalD/TARDIS is used for resource scheduling in production environmentsin several institutes within and outside of physics.5.1 Institute of Experimental Particle PhysicsAt the Institute of Experimental Particle Physics (ETP) of the Karlsruhe Insti-tute of Technology (KIT), research groups of for example the AMS, Belle, andCMS collaboration share one local HTCondor batch system. For data analysisand simulation, multiple resources are used: The normal desktop workstations, when not used interactively, and dedicatedlocal worker nodes with 600 CPU-cores are integrated in the OBS. All jobsrun in Docker-containers [8]. The HPC cluster ForHLRII at KIT with 25.000 CPU-cores is integratedusing COBalD and TARDIS. The drones are bash-scripts, backlled tothe Slurm Workload Manager, which start up a HTCondor-process. Thejobs run in a Singularity-containers [13].g OBSSubmitsjobsLocal res. 1 Local res. 2Job 1Job 2Job 3Job 4drone 1 drone 2Resource Prov. 1,Job 5Job 6drone 3 drone 4OBSplace-holderResource Prov. 2,RequestsresourcesAssessesresourcesuitabilityFig. 2. Jobs are submitted to the overlay batch-system (OBS). The OBS decides, onwhich drone to run the jobs. Depending on the metrics allocation and utilization ofthe resources, COBalD manages the amount and type of resources needed. In thedepicted example, more resources would be requested at resource provider 1. The HPC cluster BWForCluster NEMO at the university of Freiburgis also integrated using COBalD and TARDIS. It is shared with severalother communities and consists of 900 worker nodes summing up to roughly20.000 CPU-cores. The drones run as virtual machines, booted throughOpenStack via the MOAB batch system. Within the virtual machines,provided by the HPC center, the jobs run inside of Docker-containers.All resources are available to the physicists at ETP completely transparentthrough the batch system. With this setup several thousand CPU-cores are op-portunistically integrated and used as shown in g. 3.5.2 WLCG Tier 1 Center GridKaGridKa is one of the 13 tier 1 centers of the Worldwide LHC Computing Grid(WLCG), located at KIT. It provides compute and storage resources for severalhigh energy and astroparticle physics collaborations. As shown in g. 4 oppor-tunistic resources are transparently made available to the collaborations throughone dedicated compute entry-point, a so-called compute element (CE). The re-sources are managed using COBalD andTARDIS. This also allows for dynamicintegration of dedicated hardware, as e.g. graphic cards for deep learning.5.3 Fighting COVID-19The Institute of Experimental Particle Physics and the tier 1 center GridKacontribute to the distributed computing projects Rosetta@home [4] and Fold-Fig. 3. The number of CPU cores used through the OBS at ETP. The resources atthe HPC clusters NEMO and ForHLRII are dynamically integrated using COBalDand TARDIS. The number of the statically integrated CPU cores used uctuates, asthose are freed, whenever the machine is used interactively.ARC CEcloud-arc-1-kitHTCondorOBSGridKaSingle Point of EntryBonn Tier 3(BAF)KIT HPC(FORHLR2)KIT Tier 3(TOPAS)LMU MunichOpenStackBonn HPC(BONNA)Fig. 4. GridKa dynamically integrates opportunistic resources for several experimentsbehind a single point-of-entry. Jobs from the dierent experiments are only dispatchedto their corresponding resources. This happens completely transparent to the experi-ments. Figure from [10].ing@home [2] for the ght against COVID-19. Both institutes use a COBalD/TARDIS-setup with a placeholder batch-system mimicking a constant demandto allocate resources within their respective resource pools for providing themto the protein research projects. The computing payloads are pulled from thecentral management servers of the research projects and then executed on thelocal resources. With this setup, ETP and GridKa make a signicant contribu-tion. [3,5,1]6 SummaryUsing shared science computing centers, HPC clusters, and resources at cloudproviders, is a promising way to mitigate computing load peaks within researchgroups, and provide specic hardware. It also can help to achieve higher ecien-cies on those resources. To dynamically schedule and integrate heterogeneousresources on demand, COBalD and TARDIS are the necessary toolkits. Theyallow for optimizing the resource usage by reacting on the decisions of the overlaybatch-system and by this yield a more ecient resource usage.For the end-users a setup with COBalD and TARDIS allows for a transpar-ent usage of a plethora of resources through one single point-of-entry. They areno longer faced with the inconvenience and the eort of having multiple identityproviders, or several batch systems to deal with. By integrating resources pro-viding dedicated hardware, this can easily be made available for the end-usersmaintaining fair usage.References1. Fight COVID-19  Grafana (2020), https://grafana-sdm.scc.kit.edu/d/uMJtTojZk/ght-covid-192. Folding@home  Fighting disease with a world wide distributed super computer.(2020), https://foldingathome.org/3. Folding@home stats report (2020), https://stats.foldingathome.org/team/2505654. Rosetta@home (2020), https://boinc.bakerlab.org/5. Rosetta@home: KIT-GridKa (2020), https://boinc.bakerlab.org/rosetta/show_user.php?userid=21277446. Adaptive Computing: Moab HPC  Adaptive Computing (Jun 2020), https://adaptivecomputing.com/cherry-services/moab-hpc/7. Apache: Apache CloudStack: Open Source Cloud Computing (Jun 2020), http://cloudstack.apache.org/8. Docker Inc.: Empowering App Development for Developers | Docker (Jun 2020),https://www.docker.com/9. Fischer, M., Kuehn, E., Giels, M., Schnepf, M., Kroboth, S., Frey-ermuth, O.: MatterMiners/cobald: New Plugin System (Apr 2020).https://doi.org/10.5281/zenodo.3752587, https://doi.org/10.5281/zenodo.375258710. Giels, M., et al.: Eective dynamic integration and utilization of heterogenouscompute resources. Proceedings of the 24th International Conference on Comput-ing in High Energy and Nuclear Physics (2020), to be published11. Giels, M., Schnepf, M., Kuehn, E., Kroboth, S., Caspart, R., von Cube, R.F., Fis-cher, M., Wienemann, P.: MatterMiners/tardis: YAR (Yet Another Release) (Jun2020). https://doi.org/10.5281/zenodo.3874847, https://doi.org/10.5281/zenodo.387484712. HTCondor Team: HTCondor (May 2020). https://doi.org/10.5281/zenodo.3834815,https://doi.org/10.5281/zenodo.383481513. Kurtzer, G.M., Sochat, V., Bauer, M.W.: Singularity: Scientic con-tainers for mobility of compute. PLOS ONE 12(5), 120 (05 2017).https://doi.org/10.1371/journal.pone.0177459, https://doi.org/10.1371/journal.pone.017745914. OpenStack: Build the future of Open Infrastructure (Jun 2020), https://www.openstack.org/15. SchedMD: SchedMD | Slurm Support and Development (Jun 2020), https://www.schedmd.com/

Dynamic Computing Resource Extension Using COBalD/TARDIS

https://publikationen.bibliothek.kit.edu/1000124143/88960826

Dynamic Computing Resource Extension Using COBalD/TARDIS

Abstract

Similar works

Full text

Available Versions

KITopen