Cuesta Noriega, Alberto

Fernández del Castillo, Enol

González Caballero, Isidro

López García, Álvaro

Marco de Lucas, Jesús

Matorras Weinig, Francisco

Rodríguez Marrero, Ana Yaiza

Ana Y Rodríguez-Marrero

Isidro González Caballero

Alberto Cuesta Noriega

Enol Fernández-del-Castillo

Álvaro López García

Jesús Marco de Lucas

Francisco Matorras Weinig

Crossref

Journal of Physics Conference Series

Integrating PROOF Analysis in Cloud and Batch Clusters

Repositorio Institucional de la Universidad de Oviedo

This content has been downloaded from IOPscience. Please scroll down to see the full text.Download details:IP Address: 156.35.192.3This content was downloaded on 17/03/2015 at 08:34Please note that terms and conditions apply.Integrating PROOF Analysis in Cloud and Batch ClustersView the table of contents for this issue, or go to the journal homepage for more2012 J. Phys.: Conf. Ser. 396 032091(http://iopscience.iop.org/1742-6596/396/3/032091)Home Search Collections Journals About Contact us My IOPscienceIntegrating PROOF Analysis in Cloud and BatchClustersAna Y. Rodr´ıguez-Marrero1, Isidro Gonza´lez Caballero2, AlbertoCuesta Noriega2, Enol Ferna´ndez-del-Castillo1, A´lvaro Lo´pez Garc´ıa1,Jesu´s Marco de Lucas1, Francisco Matorras Weinig11Instituto de F´ısica de Cantabria, Universidad de Cantabria – CSIC, Santander, Spain2Physics Department, Universidad de Oviedo, Oviedo, SpainE-mail: arodrig@ifca.unican.esAbstract. High Energy Physics (HEP) analysis are becoming more complex and demandingdue to the large amount of data collected by the current experiments. The Parallel ROOTFacility (PROOF) provides researchers with an interactive tool to speed up the analysis of hugevolumes of data by exploiting parallel processing on both multicore machines and computingclusters. The typical PROOF deployment scenario is a permanent set of cores configured torun the PROOF daemons. However, this approach is incapable of adapting to the dynamicnature of interactive usage. Several initiatives seek to improve the use of computing resourcesby integrating PROOF with a batch system, such as Proof on Demand (PoD) or PROOFCluster. These solutions are currently in production at Universidad de Oviedo and IFCA andare positively evaluated by users. Although they are able to adapt to the computing needsof users, they must comply with the specific configuration, OS and software installed at thebatch nodes. Furthermore, they share the machines with other workloads, which may causedisruptions in the interactive service for users. These limitations make PROOF a typical use-case for cloud computing. In this work we take profit from Cloud Infrastructure at IFCA in orderto provide a dynamic PROOF environment where users can control the software configuration ofthe machines. The Proof Analysis Framework (PAF) facilitates the development of new analysisand offers a transparent access to PROOF resources. Several performance measurements arepresented for the different scenarios (PoD, SGE and Cloud), showing a speed improvementclosely correlated with the number of cores used.1. IntroductionInteractive analysis of the large data sets produced by the current High Energy Physicsexperiments is possible thanks to the parallel execution provided by PROOF [1]. This toolexecutes embarrassingly parallel analysis by distributing the work load among a set of workersdeployed as hosts in a cluster or as cores in a single multi-core computer. PROOF is designed torun the user analysis with as little modifications as possible with respect to sequential versions.However, it requires the code to be written following some conventions and requires a workingPROOF deployment that will actually run the user analysis.PROOF is primarily meant to be deployed in a cluster of machines where a master, whichcan be multi-layered, acts as entry point to the facility and distributes the work to a set ofworkers. All these machines must run a properly configured PROOF daemon. The installationand configuration of such clusters requires certain level of expertise and understanding of theInternational Conference on Computing in High Energy and Nuclear Physics 2012 (CHEP2012) IOP PublishingJournal of Physics: Conference Series 396 (2012) 032091 doi:10.1088/1742-6596/396/3/032091Published under licence by IOP Publishing Ltd 1system, mostly due to the flexibility of the PROOF system that allows it to adapt to a widerange of resource setups. Alternatively, PROOF-Lite is a dedicated version of PROOF thattakes advantage of multi-core machines, but is limited to a single host.Tools like PoD [2] or PROOF Cluster [3] free the user from the setup and configuration ofthe system by providing a dynamic construction of the PROOF infrastructure using a batchsystem. This integration of PROOF into existing production services forces the adaptation ofthe system to specific software configurations, which may not completely fulfill the requirementsof the user. Moreover, sharing the hosts with other jobs may cause a degradation of performanceand can lead to problems in the configuration of PROOF. The dynamic nature of the analysisand the need for isolation make PROOF a typical use-case for cloud computing, where virtualmachines are started on demand to create a PROOF cluster that can be accessed interactivelywith the exact software configuration required and without any interference from other jobs orusers.The PROOF Analysis Framework (PAF) [4] was created to allow users to run their analysiswith PROOF without caring about the PROOF details. The framework has two main objectives:hide the PROOF details as much as possible, so the researcher can concentrate on the analysisdevelopment and not on the setup of the infrastructure, and provide ready-to-use skeleton andtools for developing new analysis or easily migrating existing ones that can be run in parallelor sequential modes. This work describes the PAF backend that is in charge of deploying thePROOF infrastructure on behalf of the user. This backend is able to use several platforms:PROOF-Lite, batch systems (using PoD and PROOF Cluster) and Cloud.2. PROOF Analysis Framework DeploymentPAF transparently handles the automatic deployment of the PROOF cluster for the userproviding a fast response while trying to make an efficient use of the resources. PAF supportsfive different deployment modes, that the user selects according to the resources available:Sequential. This mode runs the code without any parallelization. It is intended for debuggingpurposes only and allows to isolate any potential problems caused by the analysis code fromthe PAF tools.Lite. The Lite mode starts the PROOF-Lite environment on a single machine. It allows toperform analysis of small data sets on login machines or desktops when no computer farmis available.PoD. In this case, PAF uses the PoD tools to deploy the cluster in a batch system. The masterdaemon is created dynamically in the user machine, which can cause a bottleneck whenmerging large files or a big number of users are sharing the same machine for runninganalysis.Cluster. The Cluster mode uses a fixed master configured and tuned for the specificcharacteristics of the site where it is installed. The workers however are dynamicallyconfigured and started by submitting jobs to a SGE [5] or Torque [6]/PBS [7] batch systems.Cloud. This new experimental mode creates complete PROOF clusters on demand, both masterand a set of workers, by starting virtual machines using a IaaS (Infrastructure as a Service)Cloud platform.Cluster and Cloud deployment backends are completely developed by the IFCA andUniversidad de Oviedo teams and detailed in the following sections.2.1. PAF ClusterThe PAF Cluster mode takes profit of the existing production infrastructure at IFCA andUniversidad de Oviedo. Those sites give support to both grid and local users that ultimatelyexecute their jobs at the corresponding resources, managed by a batch system scheduler.International Conference on Computing in High Energy and Nuclear Physics 2012 (CHEP2012) IOP PublishingJournal of Physics: Conference Series 396 (2012) 032091 doi:10.1088/1742-6596/396/3/0320912Under this schema, a dedicated host acts as master of the PROOF farm. This master isindependent of the batch system and is the fixed entry point that PAF uses for establishing theanalysis sessions. The workers are created by submitting jobs requesting several slots to thebatch system. This job configures and starts the daemon on all the allocated slots. Once thedaemons are started they are added to the list of workers for the user and a signal is sent toPAF, that can connect to the master to execute the user code. Data is stored in a shared filesystem mounted by all nodes. Fig. 1 shows the process flow for creating a PAF Cluster session.Figure 1. Deployment of PAF on a Batch Infrastructure.In order to improve the service level of the PAF Cluster users, the batch policies are tunedto enhance the interactive experience:• PROOF jobs are limited to two hours. This time is enough to execute most of the analysiseven with large data sets. PAF reutilizes whenever possible existing jobs within those twohours, and users are warned when only a few minutes are left. The job limitation policyenables the use of backfilling in the batch system and avoids starvation of batch jobs byfreeing resources when they are not needed anymore. Users may finalize the PAF jobs atany time by invoking the endproof command.• A range based slot allocation policy is enforced. PAF sets an upper limit for the numberof slots to allocate in a single job and the batch system using a fair use policy will adaptthe actual number of slots given to the job to the current load.• PROOF jobs have a high priority and are started within seconds from the submission. If itis not possible to allocate resources for the job immediately, batch low priority jobs runningin the cluster may be suspended.2.2. PAF CloudThe Cloud mode takes advantage of the ability to start virtual machines on demand provided byIaaS Clouds, like the pilot OpenStack [8] service at IFCA. The current implementation is basedon OpenStack and the OpenStack API, but its architecture is general enough to be adapted tosimilar public or private Cloud offerings, such as Amazon EC2 [9], with minor effort.The Cloud mode assumes that the following services are available:• A Virtual Machine (VM) Image repository, where golden copies of the VMs to be executedare stored. For the use of PAF, a VM image was created with Scientific Linux 5 and acomplete ROOT installation with the PROOF daemons.International Conference on Computing in High Energy and Nuclear Physics 2012 (CHEP2012) IOP PublishingJournal of Physics: Conference Series 396 (2012) 032091 doi:10.1088/1742-6596/396/3/0320913• A Volume server, that provides permanent storage that can be attached to the runningVMs. Data to be analyzed by the users is stored in such volumes. PAF maintains a list ofavailable data sets and the volumes where they are stored.• A Virtual Machine Orchestrator, that allows users to start, stop and manage instances ofVMs. PAF contacts this service for setting up the PROOF cluster by instantiating VMsthat are taken from the image repository. The system provides a set of VM flavors, whereeach flavor has a fixed number of CPU cores, RAM memory and disk space. The numberand characteristics of these flavors depend mostly on the capabilities underlying physicalinfrastructure.• A Network Management Services for the assignment of public IPs to running instancesand the provisioning of VPNs to connect securely to those instances. The current PAFimplementation assumes a VPN is readily available and the user machine is correctlyconfigured to use it, but it might be easily enhanced to request a public IP and assignit to the PROOF master.Fig. 2 depicts the process for deploying a PROOF cluster in the Cloud infrastructure. Oncethe user selects the Cloud mode and starts a PAF analysis the next steps are taken:Figure 2. Deployment of PAF on a Cloud Infrastructure.• A master VM is instantiated in the cloud infrastructure. Currently, this VM is instantiatedusing a flavor that has a single core and 2GB of memory.• The volume (or volumes) containing the data to be analyzed is attached to the master VM.Once attached, it is mounted in the VM.• A set of worker VMs is instantiated in the the infrastructure. Worker VM are 8-coremachines with 14GB of RAM. The number of workers to start is determined by the userand is limited to a maximum of 32 cores per session in order to avoid abuses.International Conference on Computing in High Energy and Nuclear Physics 2012 (CHEP2012) IOP PublishingJournal of Physics: Conference Series 396 (2012) 032091 doi:10.1088/1742-6596/396/3/0320914• A NFSv4 server that exports the data from the volumes is configured and started in themaster VM. This filesystem is mounted at all the worker VMs and data becomes availableat all the VMs.• The PROOF cluster is created by building a configuration file that contains both the masterand worker VMs, and then starting the daemons at each VM. The system is ready foraccepting connections from the clients. The IP of the master is returned to PAF and theanalysis session can start.Users can reuse their PROOF infrastructure during several sessions. Once they have finishedall their analysis tasks, an endproof script terminates the instances and frees all the resources.3. PAF in ProductionUsers have incorporated PAF in their daily analysis routine using Cluster and PoD modes onthe computing infrastructures at IFCA and Universidad de Oviedo respectively. Both centersprovide resources for the Spanish participation at the LHC-CMS experiment [10]. IFCA usesthe SGE batch system for managing more than 2000 cores. All the nodes in the cluster mounta GPFS [11] shared file system that contains user homes and the CMS data. Universidad deOviedo has a 100 cores cluster managed by Torque and a Hadoop [12] filesystem for the data.The Cloud Pilot at IFCA is currently at a testing phase for advanced users and consists of aOpenStack service which manages 8-core hosts configured with the Xen hypervisor for runningthe Virtual Machines. No shared filesystem is provided by the infrastructure, but as shownpreviously, a NFS server is used for sharing the data between the PROOF VMs.PAF is used as the main framework for the final stages of the physics analysis carried out bythe researchers involved in the CMS Collaboration at IFCA and Universidad de Oviedo. Fig.3 shows the performance of a PAF analysis on real data sets from the CMS top physics group.Two different samples of events, with 19 GB and 38 GB were used, for each of the computinginfrastructures available: PAF Cluster at the SGE production system at IFCA (labeled asIFCA), PAF PoD at the Universidad de Oviedo Torque cluster (labeled as Oviedo) and PAFCloud using the OpenStack resources at IFCA (labeled as Cloud). The scalability of the analysisis limited by the I/O bandwidth and no performance gains are found beyond 20 nodes. Users aresatisfied with the time reduction of their executions, that allows them to analyze big amountsof data in a few minutes.Figure 3. Time spent (left) and speedup factor (right) using PAF over a different number ofworkers.International Conference on Computing in High Energy and Nuclear Physics 2012 (CHEP2012) IOP PublishingJournal of Physics: Conference Series 396 (2012) 032091 doi:10.1088/1742-6596/396/3/03209154. ConclusionsWe have developed a framework based on PROOF to facilitate the parallel execution of theresearchers codes taking profit from the existing computing infrastructures at the sites. Theframework handles transparently the automatic and dynamic deployment of the PROOF systemfor the user in different types of resources. PAF provides an environment for performinginteractive analysis of large data sets using the production infrastructures of IFCA andUniversidad de Oviedo. Private cloud services, like the OpenStack installation at IFCA, canalso be accessed using PAF thanks to the Cloud mode that creates the PROOF cluster on topof virtual machines.CMS local users at both institutions have been using PAF since its early integration, andthey are satisfied with the facility. The outcome of the analysis using PAF have ended in thepublication of results.References[1] Ballintijn M, Biskup M, Brun R, Canal P, Feichtinger D, Ganis G, Kickinger G, Peters A and Rademakers F2006 Nucl. Inst. & Meth. in Phys. Res. A 559 13–16[2] Malzacher P and Manafov A 2010 J. Phys.: Conf. Ser. 219 072009[3] Rodr´ıguez Marrero A Y, Gonza´lez Caballero I, Cuesta Noriega A and Matorras Weinig F 2011 J. Phys.: Conf.Ser. 331 072061[4] Gonza´lez Caballero I, Rodr´ıguez Marrero A, Ferna´ndez del Castillo E and Cuesta Noriega A 2012 A PROOFanalysis framework, to be published at the Proc. for CHEP2012[5] Son of Grid Engine (SGE): https://arc.liv.ac.uk/trac/SGE[6] TORQUE: http://www.adaptivecomputing.com/products/open-source/torque[7] Portable Batch System (PBS): http://www.pbsworks.com/Product.aspx?id=1[8] OpenStack: http://openstack.org[9] Amazon Elastic Compute Cloud (EC2): http://aws.amazon.com/ec2[10] The CMS Collaboration 2008 J. Inst. 3 S08004[11] IBM General Parallel File System (GPFS): http://www.ibm.com/systems/software/gpfs[12] Hadoop distributed file system: http://hadoop.apache.orgInternational Conference on Computing in High Energy and Nuclear Physics 2012 (CHEP2012) IOP PublishingJournal of Physics: Conference Series 396 (2012) 032091 doi:10.1088/1742-6596/396/3/0320916

Integrating PROOF analysis in cloud and batch clusters

http://digibuo.uniovi.es/dspace/bitstream/10651/18963/1/1742-6596_396_3_032091.pdf

Integrating PROOF analysis in cloud and batch clusters

Abstract

Similar works

Full text

Available Versions

Crossref

Repositorio Institucional de la Universidad de Oviedo