Compute clusters use Portable Batch Systems (PBS) to distribute workload among individual cluster machines. To extend standard batch systems to Cloud infrastructures, a new service monitors the number of queued jobs and keeps track of the price of available resources. This meta-scheduler dynamically adapts the number of Cloud worker nodes according to the requirement profile. Two different worker node topologies are presented and tested on the Amazon EC2 Cloud service

Baun, C.

Büge, V.

Hauth, T.

Kunze, M.

Quast, G.

Scheurer, A.

English

KITopen

Journal of Physics: Conference SeriesDynamic Extensions of Batch Systems with CloudResourcesTo cite this article: T Hauth et al 2011 J. Phys.: Conf. Ser. 331 062034 View the article online for updates and enhancements.Related contentDynamic Extension of a Virtualized Clusterby using Cloud ResourcesOliver Oberst, Thomas Hauth, DavidKernert et al.-Integration of virtualized worker nodes instandard batch systemsVolker Büge, Hermann Hessling, YvesKemp et al.-Cloud Bursting with GlideinWMS: Meansto satisfy ever increasing computing needsfor Scientific WorkflowsParag Mhashilkar, Anthony Tiradani, BurtHolzman et al.-Recent citationsOn-demand provisioning of HEP computeresources on cloud sites and shared HPCcentersG Erli et al-Dynamic provisioning of a HEP computinginfrastructure on a shared hybrid HPCsystemKonrad Meier et al-Dynamic Extension of a Virtualized Clusterby using Cloud ResourcesOliver Oberst et al-This content was downloaded from IP address 129.13.72.197 on 14/11/2018 at 12:09Dynamic Extensions of Batch Systems with CloudResourcesT Hauth1, G Quast1, M Kunze2, V Büge1, A Scheurer1 and C Baun21 Karlsruhe Institute of Technology (KIT), Institut für Experimentelle Kernphysik (EKP),Campus Süd, Postfach 69 80, 76128 Karlsruhe, Germany2 Karlsruhe Institute of Technology (KIT), Steinbuch Centre for Computing (SCC),Hermann-von-Helmholtz-Platz 1, 76344 Eggenstein-Leopoldshafen, GermanyE-mail: hauth@ekp.uni-karlsruhe.deAbstract. Compute clusters use Portable Batch Systems (PBS) to distribute workload amongindividual cluster machines. To extend standard batch systems to Cloud infrastructures, a newservice monitors the number of queued jobs and keeps track of the price of available resources.This meta-scheduler dynamically adapts the number of Cloud worker nodes according to therequirement profile. Two different worker node topologies are presented and tested on theAmazon EC2 Cloud service.1. IntroductionHigh Performance Computing infrastructures at research centers and universities aretraditionally maintained by different user groups or in a centralized way by the computingdepartment. The latter intrinsically profits from the existing infrastructure and expertknowledge combined with load balancing between different groups. Assuming a traditionalcluster setup, the price for this is a common operating system infrastructure for all users. Herevirtualization of operating systems is a promising technique to overcome this drawback as itallows deploying the needed number of fully customized worker nodes on a short time scale.Although Cloud Computing is well accepted for the deployment of service machines, it is alsoan ideal technique for a dynamic allocation of batch system worker nodes. In the following, anextension of standard batch systems to allocate Cloud resources dynamically in the context ofHigh Energy Physics computing is presented, supporting both the Ubuntu Enterprise Cloud [1]using Eucalyptus [2] as back end and Amazon Elastic Computing Cloud (EC2) [3].1.1. Portable Batch System (PBS)Traditionally, cluster based computing infrastructures employ a PBS to distribute computingjobs among cluster machines called worker nodes. Users submit their jobs to batch queuesholding them until resources are available and the jobs can be handed over to worker nodes.Typical installations use a static list of worker nodes within the local data center. This listrarely changes and nodes are only removed for maintenance or due to hardware failures.A popular open-source implementation of a PBS is the TORQUE resource manager by ClusterResources [4] which is used and maintained by the scientists of the EKP [5] in a local computingcluster.International Conference on Computing in High Energy and Nuclear Physics (CHEP 2010) IOP PublishingJournal of Physics: Conference Series 331 (2011) 062034 doi:10.1088/1742-6596/331/6/062034Published under licence by IOP Publishing Ltd 11.2. Cloud ComputingThe term Cloud Computing covers a large variety of applications. This paper refers to theInfrastructure-as-a-Service (IaaS) paradigm. Cloud providers offer the possibility to run pre-packed or custom virtual machine images on physical machines within their data center. Incontrast to cluster based computing infrastructures, it is quite common to boot up many Cloudinstances for a short amount of time and to discard them, as soon as they are not needed anymore.During the last years, various Cloud providers and implementations entered the servicemarket. One of the first to offer Cloud services was Amazon with EC2. They offer differentmachine configurations, so called machine types, on a pay-per-hour basis. The software usedby Amazon to provide and manage their Cloud services is proprietary and not available to thepublic.The company Eucalyptus Systems provides a solution to fill this gap with an open-sourceCloud Computing infrastructure software, called Eucalyptus, which implements the same APIas Amazon’s EC2 does.The Cloud Computing research group at the Steinbuch Centre for Computing [6] at theKarlsruhe Institute of Technology runs a private Cloud based on Eucalyptus. At its currentstage of expansion, the private Cloud can run up to 40 single-core virtual machines.2. Extending Local Infrastructures with Cloud ResourcesThe EKP institute is taking an active role in processing and analyzing data sampled by the LargeHadron Collider’s (LHC) Compact Muon Solenoid (CMS) physics experiment at CERN [7]. Aspart of this research project, scientists at EKP have to utilize as many computing resourcesas needed to get the analysis jobs done in time. Up to now, a cluster PBS handles this load.Due to the static nature of this installation, it is not possible to add more computing resourcesif needed. Especially during peek demands, the utilization of Cloud resources to extend thelocal computing capacity is desired. Traditional cluster setups do not offer this kind of dynamicextendibility.Moreover, the operating system and application setup of the local research groups arespecifically tuned to meet the needs of the CMS experiment. In addition, a second clustershared among nine different institutions could be used, but the cluster setup does not cover thespecific operating system and software needs of CMS. All users of the cluster have to run theirapplications on the same software stack. Related research of the EKP on batch virtualization[8]provides a solution to overcome this drawback on traditional clusters.Cloud Computing allows to customize the software setup down to the operating systemand multiple virtual machine images can be maintained and the one best suited for a specificcomputation job can be booted. Moreover, since many Cloud sites by different companiesare available, the Cloud provider can be changed in an easy fashioned way or more than oneprovider can be used simultaneously. This allows to leverage the different Cloud provider’spayment models and react on pricing changes.3. ROCED: A Cloud Meta-SchedulerIn order to profit from the advantages discussed above, a prototype meta-scheduler calledROCED ( Responsive on-demand Cloud Enabled Deployment ) was developed and featuresa modular design to provide a flexible way to assess different concepts and technologies.3.1. Design IdeaThe ROCED scheduler processes various input parameters and decides about the utilization ofthe connected Cloud resources. The following input parameters are defined:International Conference on Computing in High Energy and Nuclear Physics (CHEP 2010) IOP PublishingJournal of Physics: Conference Series 331 (2011) 062034 doi:10.1088/1742-6596/331/6/0620342ROCED CoreRequirement Adapters... supplies information about needed compute nodes, e.g. PBS queue sizeTorque Adapter...Integration Adapters... integrates booted compute nodesinto existing PBS ServerTorque Adapter...Site Adapters... boot machines on various CloudComputing sitesAmazon EC2Eucalyptus...Broker... decides which machines to boot or shutdownFigure 1. ROCED’s modular design allows for the replacement of any component: PBS(Integration and Requirement Adapters), Cloud back end (Site Adapters).• Current pricing of Cloud resources• Current workload on the batch system• Priority of certain processing jobs• Number of overall worker nodes• Manual decision of the administrator to increase or decrease the computing power of thesystemDepending on pre-defined rules and the current requirement profile, ROCED will decidewhich Cloud sites to use and how many instances to start inside the Cloud. In the same step,it will shut down instances which are no longer needed in order to free Cloud resources. Sincemany Cloud providers are following the principle pay-as-you-go on an hourly basis, this allowsto keep the costs as low as possible.Once the Cloud worker nodes are started and running, ROCED integrates them into thebatch server’s worker nodes listing. Therefore an OpenVPN [9] tunnel is established betweenthe Cloud instance and the local network. Afterwards, the new worker nodes are registered atthe batch server and jobs are being transferred for processing.If ROCED decides that a specific worker node is no longer needed within the batch system,this node is excluded from the list of active nodes. As soon as the node finished its remainingjobs, it is removed from the PBS and the Cloud instance is stopped.This guided way of node registration and deregistration ensures that the operations of thePBS server are not disturbed by adding and removing worker nodes.3.2. ImplementationROCED is implemented using Python 2.6. All major components of the system are managed byso-called Adapters which supply a concrete implementation of various abstract functionalities(Figure 1).So far, ROCED supports the Amazon EC2 and Eucalyptus Cloud interfaces and theTORQUE resource manager. Other batch servers and Cloud providers can be seamlesslyintegrated into the application by programming new Adapters. The Broker component, whichdecides about the Cloud usage, will then take the new resources into consideration.3.3. Cloud Worker Node TopologyTwo different types of topologies to attach Cloud worker nodes to a local batch server wereinvestigated and can be set up in a dynamic way using ROCED.International Conference on Computing in High Energy and Nuclear Physics (CHEP 2010) IOP PublishingJournal of Physics: Conference Series 331 (2011) 062034 doi:10.1088/1742-6596/331/6/06203433.3.1. Topology 1: Route jobs to a specific Cloud PBS. To support a large amount of workernodes it is preferred to boot a second, dedicated PBS server within the Cloud data center.Once ROCED has booted new Cloud worker nodes, they will be added to the Cloud PBS. Thisnodes can use the internal network infrastructure of the data center for the communicationwith the PBS server. The internal network traffic is free of charge for most Cloud providers.Jobs submitted to the local PBS are routed to the Cloud PBS. The TORQUE installation wasconfigured to provide a separate queue cloud which forwards all jobs to the Cloud PBS.This approach takes more effort to set up, but the benefit is, that the PBS communicationwill not leave the Cloud data center.3.3.2. Topology 2: Attach Cloud worker nodes to local PBS directly. As soon as the Cloudworker node is started and the OpenVPN tunnel is established between the node and the localnetwork, the node name and the VPN network IP address are added to the local PBS machine.All node status reports and job information are sent between node and PBS server via the VPN.This allows an easy addition of new Cloud worker nodes to the local PBS since all Cloud specificsare abstracted by the VPN tunnel.As the number of Cloud worker nodes increases, the communication between nodes and serverwill require higher networking bandwidth. This can exhaust the bandwidth of the VPN tunnelwhich will degrade the performance of other networking services. In conclusion, this topology isfeasible for small to medium sized installations.4. Setting Up Topology 1: Routing Local Jobs to Cloud PBS4.1. Test SetupA Linux distribution that suits the needs of particle physics applications is Scientific Linux [10].In order to generate a Scientific Linux 5.4 x86-64 machine image with the size of 4 GB and toupload it to the Amazon Cloud, the supplied Amazon toolkit was used. The only additionalsoftware installed was the TORQUE worker node executable for 64-bit architecture (TORQUEVersion 2.4.6). This machine image will act as the worker node.The machine providing the TORQUE server was run within the Amazon EC2 Cloud site.For that purpose, an Ubuntu 9.10 AMD64 image and TORQUE 2.4.7 was installed from source.A batch queue was created and configured so each worker node will calculate one batch job ata time.A queue named cloud was created on the local TORQUE server and configured to route allincoming jobs to the Cloud PBS.The ROCED meta-scheduler itself ran on an Ubuntu 9.10 64-bit desktop machine withinthe local network and monitored the batch queue of the TORQUE server in the Cloud. Themaximum number of Cloud instances ROCED was allowed to start on EC2 was set to 19, soincluding the Cloud PBS machine, 20 virtual machines were running at the most.4.2. Test RunTo stress test ROCED and the PBS server-to-node communication within the Cloudenvironment, 1000 jobs were submitted, each running for a period of 10 seconds. Such shortjob run time was chosen so the batch server has to transfer jobs to its nodes very often and thusincreasing the overall load on the batch system.As seen in Figure 2, as soon as the jobs appear in the batch queue, ROCED immediatelystarts the maximum of 19 possible Cloud instances. It takes 130 seconds until the first machine isbooted and integrated into the PBS. After 300 seconds all 19 machines are booted and integratedinto the PBS. As soon as the batch queue is depleted, ROCED starts to remove worker nodesfrom the PBS and terminates the Cloud instances.International Conference on Computing in High Energy and Nuclear Physics (CHEP 2010) IOP PublishingJournal of Physics: Conference Series 331 (2011) 062034 doi:10.1088/1742-6596/331/6/0620344Figure 2. 1000 batch jobs added to the batch queue. Note that Jobs in batch queue is scaledby 0.02.In conclusion, it is shown that a large number of jobs can be transferred to the Cloud PBS,ROCED is able to dynamically add the needed worker nodes and the PBS can leverage the newcomputing resources immediately.5. Setting Up Topology 2: Attaching Cloud Worker Nodes to Local PBSThe next example presented is the possibility to extend a local PBS, which already has a numberof static worker nodes attached. As soon as the local batch queue holds more than a specifiedamount of jobs, the so called Cloud threshold, new Cloud worker nodes are allocated. This canmultiply the batch system’s computing capacity. The system configuration is equal to the setupin the first example, except for the TORQUE and ROCED setup.5.1. Test SetupOne worker node residing in the local network is statically attached to the PBS system. NewCloud worker nodes are connected via a VPN tunnel to the local network and added to theworker node list of the local PBS server. All test jobs are submitted to the queue of the localbatch server and each worker node is configured to process two jobs at a time.ROCED monitors the size of the queue on the local PBS and is configured to start Cloudworker nodes as soon as the queue size exceeds 8 jobs (Cloud threshold = 8). ROCED wasconfigured to start 4 Cloud worker node instances at maximum.5.2. Test RunTo test the adaptiveness of ROCED’s Cloud usage, a varying number of jobs were sent the toPBS where each job runs for 200 seconds. (Figure 3).As the first 8 jobs are submitted to the batch server, 2 are instantly handed to the staticworker node. The other 6 jobs are left in the queue. If 8 more jobs are added, the Cloudthreshold of 8 is exceeded and ROCED starts and integrates 3 out of 4 Cloud worker nodesrequired to empty the queue. If more jobs appear in the queue, ROCED starts all available 4Cloud worker nodes thus increasing the computing power of the local PBS by a factor of five.In conclusion, it was shown that ROCED is able to increase the computing power of a localPBS outfitted with static worker nodes in a dynamic and requirement-based way.International Conference on Computing in High Energy and Nuclear Physics (CHEP 2010) IOP PublishingJournal of Physics: Conference Series 331 (2011) 062034 doi:10.1088/1742-6596/331/6/0620345Figure 3. A varying number of jobs is added to the batch queue. ROCED will only start newCloud worker nodes if the batch queue holds more than 8 jobs (Cloud threshold).6. Conclusion, Outlook and Future WorkBy using a meta-scheduler, it is possible to extend the computing resources of a standard batchsystem using Cloud Computing resources of different providers. The PBS system was madeCloud-aware without disturbing regular operations.Two Cloud worker node topologies were introduced and the advantages and drawbacks wereevaluated. Attaching Cloud worker nodes to a local PBS server is easy to set up, but resultsin an increased traffic between the Cloud data center and the local network. Routing jobs to acloud based PBS is more complex to set up, but has the advantage that some of the PBS networktraffic is limited to the Cloud data center. A test installation using ROCED demonstrated thefeasibility of both approaches incorporating TORQUE and the Amazon EC2 service.Next steps are the extension of the EKP’s production cluster with ROCED and Cloudresources in order to handle peak loads on the system - so called Cloud bursting.AcknowledgementsWe want to thank all involved EKP and SCC members of KIT Campus North and South fortheir support and help during the development and implementation of ROCED. In addition wethank the ”Bundesministerium für Bildung und Forschung” (BMBF) as well as the HelmholtzAlliance ”Physics at the Terascale” for their financial supportReferences[1] Canonical http://www.ubuntu.com/cloud[2] Eucalyptus http://open.eucalyptus.com[3] Amazon EC2 http://aws.amazon.com/ec2/[4] Cluster Resources http://www.clusterresources.com[5] EKP http://www-ekp.physik.uni-karlsruhe.de[6] SCC http://www.scc.kit.edu[7] CERN http://www.cern.ch[8] Büge V, Hessling H, Kemp Y, Kunze M, Oberst O, Quast G, Scheurer A and Synge O 2010 Journal ofPhysics: Conference Series 219 052010 URL http://stacks.iop.org/1742-6596/219/i=5/a=052010[9] OpenVPN http://openvpn.net[10] Scientific Linux https://www.scientificlinux.orgInternational Conference on Computing in High Energy and Nuclear Physics (CHEP 2010) IOP PublishingJournal of Physics: Conference Series 331 (2011) 062034 doi:10.1088/1742-6596/331/6/0620346

Dynamic extensions of batch systems with cloud resources

https://publikationen.bibliothek.kit.edu/1000043420/19379481

Dynamic extensions of batch systems with cloud resources

Abstract

Similar works

Full text

Available Versions

KITopen