The CMS experiment collects and analyzes large amounts of data coming from high energy particle collisions produced by the Large Hadron Collider (LHC) at CERN. This involves a huge amount of real and simulated data processing that needs to be handled in batch-oriented platforms. The CMS Global Pool of computing resources provide +100K dedicated CPU cores and another 50K to 100K CPU cores from opportunistic resources for these kind of tasks and even though production and event processing analysis workflows are already managed by existing tools, there is still a lack of support to submit final stage condor-like analysis jobs familiar to Tier-3 or local Computing Facilities users into these distributed resources in an integrated (with other CMS services) and friendly way. CMS Connect is a set of computing tools and services designed to augment existing services in the CMS Physics community focusing on these kind of condor analysis jobs. It is based on the CI-Connect platform developed by the Open Science Grid and uses the CMS GlideInWMS infrastructure to transparently plug CMS global grid resources into a virtual pool accessed via a single submission machine. This paper describes the specific developments and deployment of CMS Connect beyond the CI-Connect platform in order to integrate the service with CMS specific needs, including specific Site submission, accounting of jobs and automated reporting to standard CMS monitoring resources in an effortless way to their users

Aftab Khan, F.

Balcas, J.

Bockelman, B.

Gardner, R., Jr.

Hurtado Anampa, K.

Jayatilaka, B.

Lannon, K.

Larson, K.

Letts, J.

Marra Da Silva, J.

Mascheroni, M.

Mason, D.

Tiradani, A.

Yzquierdo, A. Perez-Calero

English

DigitalCommons@University of Nebraska

University of Nebraska - LincolnDigitalCommons@University of Nebraska - LincolnFaculty Publications, Department of Physics andAstronomy Research Papers in Physics and Astronomy2017CMS ConnectJ. BalcasB. BockelmanR. Gardner Jr.K. Hurtado AnampaB. JayatilakaSee next page for additional authorsFollow this and additional works at: https://digitalcommons.unl.edu/physicsfacpubThis Article is brought to you for free and open access by the Research Papers in Physics and Astronomy at DigitalCommons@University of Nebraska -Lincoln. It has been accepted for inclusion in Faculty Publications, Department of Physics and Astronomy by an authorized administrator ofDigitalCommons@University of Nebraska - Lincoln.AuthorsJ. Balcas, B. Bockelman, R. Gardner Jr., K. Hurtado Anampa, B. Jayatilaka, F. Aftab Khan, K. Lannon, K.Larson, J. Letts, J. Marra Da Silva, M. Mascheroni, D. Mason, A. Perez-Calero Yzquierdo, and A. TiradaniJournal of Physics: Conference SeriesPAPER • OPEN ACCESSCMS ConnectTo cite this article: J Balcas et al 2017 J. Phys.: Conf. Ser. 898 082032 View the article online for updates and enhancements.Related contentEnabling opportunistic resources for CMSComputing OperationsD. Hufnagel and the CMS Collaboration-Stability and scalability of the CMS GlobalPool: Pushing HTCondor and glideinWMSto new limitsJ Balcas, B Bockelman, D Hufnagel et al.-Recent Standard Model results from CMSSimon de Visscher and CMS collaboration-This content was downloaded from IP address 129.93.168.10 on 04/08/2019 at 21:181Content from this work may be used under the terms of the Creative Commons Attribution 3.0 licence. Any further distributionof this work must maintain attribution to the author(s) and the title of the work, journal citation and DOI.Published under licence by IOP Publishing Ltd1234567890CHEP IOP PublishingIOP Conf. Series: Journal of Physics: Conf. Series 898 (2017) 082032  doi :10.1088/1742-6596/898/8/082032CMS ConnectJ Balcas1, B Bockelman2, R Gardner Jr.3, K Hurtado Anampa4, BJayatilaka5, F Aftab Khan6, K Lannon4, K Larson5, J Letts7, J MarraDa Silva8, M Mascheroni5, D Mason5, A Perez-Calero Yzquierdo9,10and A Tiradani51 California Institute of Technology, Pasadena, CA, USA2 University of Nebraska-Lincoln, Lincoln, NE, USA3 University of Chicago, Chicago, IL, USA4 University of Notre Dame, Notre Dame, IN, USA5 Fermi National Accelerator Laboratory, Batavia, IL, USA6 National Centre for Physics, Quaid-I-Azam University, Islamabad, Pakistan7 University of California, San Diego, La Jolla, CA, USA8 Universidade Estadual Paulista, Sa˜o Paulo, Brazil9 Port d’Informacio´ Cient´ıfica, Barcelona, Spain10 Centro de Investigaciones Energe´ticas Medioambientales y Tecnolo´gicas, Madrid, SpainE-mail: khurtado@nd.eduAbstract. The CMS experiment collects and analyzes large amounts of data coming from highenergy particle collisions produced by the Large Hadron Collider (LHC) at CERN. This involvesa huge amount of real and simulated data processing that needs to be handled in batch-orientedplatforms. The CMS Global Pool of computing resources provide +100K dedicated CPU coresand another 50K to 100K CPU cores from opportunistic resources for these kind of tasks andeven though production and event processing analysis workflows are already managed by existingtools, there is still a lack of support to submit final stage condor-like analysis jobs familiar toTier-3 or local Computing Facilities users into these distributed resources in an integrated (withother CMS services) and friendly way. CMS Connect is a set of computing tools and servicesdesigned to augment existing services in the CMS Physics community focusing on these kind ofcondor analysis jobs. It is based on the CI-Connect platform developed by the Open Science Gridand uses the CMS GlideInWMS infrastructure to transparently plug CMS global grid resourcesinto a virtual pool accessed via a single submission machine. This paper describes the specificdevelopments and deployment of CMS Connect beyond the CI-Connect platform in order tointegrate the service with CMS specific needs, including specific Site submission, accounting ofjobs and automated reporting to standard CMS monitoring resources in an effortless way totheir users.1. IntroductionThe Compact Muon Solenoid (CMS) is a high energy physics experiment analyzing proton-proton and heavy ion collisions produced at the Large Hadron Collider (LHC) at CERN. Thestudy of the Higgs boson and the Standard model, the search for supersymmetric particles, extradimensions and particles that could make up dark matter are some of the topics this detectorhas been designed to investigate [1].However, the searches for new physics are not the only challenges facing the experiment. TheLHC generated 75 Petabytes of data during its first three years of running and it is generating21234567890CHEP IOP PublishingIOP Conf. Series: Journal of Physics: Conf. Series 898 (2017) 082032  doi :10.1088/1742-6596/898/8/082032Figure 1. Offline data processing: Production and Analysis jobs are handled differently andboth have their own submission tools or services to access the CMS Global Pool.data at a rate of 40-50 Petabytes per year during its current second run [2]. Even after processing,selection and reduction of reconstructed data carried out centrally in CMS, a single physicsanalysis can still require from dozens to hundreds of terabytes of data and there is a particularhigh demand for computing resources for data processing in the experiment because of this. TheCMS Global Pool [3] is the main source of computing resources to fulfill this need.Different tools exist to submit jobs for simulation, reconstruction and analysis of events tothe Global Pool. For instance, the production of simulated events is performed by a centraloperations team with the WMAgent framework [4], while analysis jobs based on the standardCMS executable (cmsRun) are submitted by individual users, mostly using the CMS RemoteAnalysis Builder (CRAB) [5], as seen in Figure 1. However, a variety of physics workflows goingfrom generation of simulated events to the analysis of the detector data, background studies,creation of histograms and plotting do not depend on cmsRun and often either provide supportfor different batch systems or are easier to handle and submit via bare HTCondor jobs. DataAnalysis Facilities like the LPC CAF at Fermilab [6] or Tier-3 sites are generally used for theseworkflows, but such alternatives are only valid to run locally.CMS Connect was developed to provide users the ability to submit HTCondor jobs to CMSGlobal Pool resources on a submission interface that is similar to the analysis facilities thatphysicists are used to work with, yet fully integrated with CMS submission infrastructurecomponents to guarantee proper fair share of resources per user, as well as reporting analysisactivity to standard CMS monitoring services.The sections below describe the platform CMS Connect was built on and the differentexisting components in the service that provide user account creation, group management, jobsubmission, reporting of jobs and the service monitoring and notification system.2. Base platformCMS Connect is built on CI Connect [7], a platform developed for building multi-institutionalvirtual cluster environments. This platform relies on different well supported technologies to31234567890CHEP IOP PublishingIOP Conf. Series: Journal of Physics: Conf. Series 898 (2017) 082032  doi :10.1088/1742-6596/898/8/082032Figure 2. Diagram of the CI Connect platform.provide a set of computing services focused on batch oriented analysis. CI Connect has beenadopted by different organizations and institutions, such as the Open Science Grid [8], ATLAS[9] and Duke University [10].The different elements of this platform can be seen in Figure 2. A web portal integrated withNexus [11], a Globus service to provide identity and group management, is used to register andorganize users in different group projects. A user account to the login node is auto provisionedafter approval, giving the user interactive login access to a submission point with HTCondor,access to distributed software through CVMFS [12] and access to data through the wide-areanetwork with XRootD [13].CI Connect services are hosted in a 100 Gbps Science DMZ, a scalable network architectureexplicitely designed for high-performance scientific applications [14].3. CMS Connect ComponentsAs illustrated in Figure 3, CMS Connect starts with the CI Connect elements that managethe user account creation and storage configuration, followed by the components that provideintegration with the CMS Global Pool and perform job reporting. Each one of these componentsis described below.3.1. Account creation and group managementUser identities are managed by Globus Nexus, as previously mentioned in Section 2. Nexusrequires only minimal information about the user (username, Full Name, Institution, emailaddress and password), verifies password strength, username uniqueness and provides a groupinterface that allows organization of users per institution, as shown in Figure 4. Managers cancreate analysis subgroups allowing for better organization and monitoring of activity in theservice.41234567890CHEP IOP PublishingIOP Conf. Series: Journal of Physics: Conf. Series 898 (2017) 082032  doi :10.1088/1742-6596/898/8/0820323.2. Stash AreaEven though all CMS users have access to storage for analysis at either Tier-2 sites or CentralAnalysis Facilities, it is sometimes convenient to have storage on the submission machine forcertain activities.CMS Connect users have access to a temporary data storage provided by the Open ScienceGrid to assist with pre-stage job input files, sandboxes, etc. This area can be accessed via fusein the login node, or XRootD and HTTP remotely. The latter can be used as an alternativeto HTCondor transfer mechanisms to avoid adding more work and network activity to thesubmission machine job scheduler.3.3. Job submissionHTCondor is used at the login node to allow users to submit jobs. Additional components areadded in a wrapper that runs on top of HTCondor in order to integrate the job scheduler withthe CMS Global Pool.Users are required to send their CMS proxy certificates with their jobs. This allows the serviceto preserve fair share accounting in the pool by mapping the user’s Distinguished Name (DN)with its corresponding CERN username (which is used for such accounting). This requirementalso increases the security and traceability in the submission system and avoids issues withworkflows that need to transfer data from and to other CMS Tier sites (CMS proxy certificatesare mandatory for this).CMS Connect submits to all computing resources available at the Global Pool unless theuser specifies a list of desired sites, which is done by adding a single HTCondor ClassAd (amapping from attribute names to expressions) in the submit file [15]. Such ClassAd is laterrecognized by the GlideinWMS infrastructure in the Global Pool and handled properly forresource provisioning.Figure 3. CMS Connect main components.Figure 4. Group Management in CMS Connect.51234567890CHEP IOP PublishingIOP Conf. Series: Journal of Physics: Conf. Series 898 (2017) 082032  doi :10.1088/1742-6596/898/8/082032Figure 5. Diagram of the CMS Dashboard integration.3.4. CMS DashboardThe CMS Dashboard [16] is the service where the experiment archives production and analysisactivity on the computing resources. CMS Connect integrates dashboard reporting forHTCondor jobs submitted from this service.The dashboard reporting procedure is pictured in Figure 5. When users submit jobs, threeparameter IDs required by the dashboard are generated to report the task project. Suchparameters are added to the job ClassAds and worker nodes inherit such Ids in their environment.Each job status is then reported from their respective worker node through a wrapper. When ajob is deleted, CMS Connect reads the dashboard ClassAds in the job to report it as killed.A CMS dashboard pie chart of jobs completed via CMS Connect and the different sites suchjobs were completed at can be seen Figure 6.4. Service monitoring and notification systemMaintaining the high availability of a complex system is often a challenge, as the failure of asingle component can affect the proper functioning of the entire service. Therefore, monitoringthese components is extremely important.CMS Connect uses Jenkins [17], an open source automation server to periodically test allservice components, including access to the login node, Stash and CVMFS availability andsubmission of jobs to the CMS Global Pool, as shown in Figure 7. When one of these componentsfails, an email notification is immediately sent to the CMS Connect system administrator.Additionally, failures related to the CI Connect platform can be notified to the Open ScienceGrid help desk for the right support team to take action, allowing us to maximize the availabilityof the service to the users.5. ConclusionsWe have developed a service designed to complement existing submission tools in theCMS experiment for HTCondor-based analysis workflows. CMS Connect provides a virtualenvironment that resembles local cluster submission interfaces, but has all advantages the CMSGlobal Pool has to offer, besides providing integration with the CMS dashboard for reporting.61234567890CHEP IOP PublishingIOP Conf. Series: Journal of Physics: Conf. Series 898 (2017) 082032  doi :10.1088/1742-6596/898/8/082032Figure 6. CMS Dashboard.Figure 7. CMS Connect service monitoring via Jenkins.AcknowledgmentsThe Open Science Grid and University of Chicago Team not only provide and maintain theCI-Connect based submission hardware and networking infrastructure, but also support theservice on different levels, including hosting, registration, software updates and maintenancenotifications, within other tasks. We are especially grateful to Robert Gardner, LincolnBryant, Suchandra Thapa and Balamurugan Desinghu who assisted closely in the integrationof the CI-Connect platform with the CMS Global Pool and continue supporting the serviceoperability. The present work is partially funded under grants from the U.S Department ofEnergy, the National Science Foundation and Spain Ministry of Economy and Competitiveness71234567890CHEP IOP PublishingIOP Conf. Series: Journal of Physics: Conf. Series 898 (2017) 082032  doi :10.1088/1742-6596/898/8/082032grant FPA2013-48082-C2-1/2-R. The Port d Informacio´ Cient´ıfica (PIC) is maintained througha collaboration between the Generalitat de Catalunya, CIEMAT, IFAE and the UniversitatAutonoma de Barcelona.References[1] Chatrchyan S et al. (The CMS Collaboration) 2008 The CMS experiment at the CERN LHC JINST 3 S08004[2] Cancio G, Bahyl V, Kruse D, Leduc J, Cano E and Murray S 2015 Experiences and challenges running CERN’shigh capacity tape archive J. Phys. Conf. Ser. 664 042006[3] Balcas J et al. 2016 Stability and scalability of the CMS Global Pool: Pushing HTCondor and glideinWMSto new limits, to be published in these proceedings[4] Boudoul G, Franzoni G, Norkus A, PolA , Srimanobhas P and Vlimant J R 2015 Monte Carlo ProductionManagement at CMS J. Phys. Conf. Ser. 664 072018[5] Cinquilli M, Spiga D, Grandi C, Hernandez J M, Konstantinov P, Mascheroni M, Riahi H and VaanderingE 2012 CRAB3: Establishing a new generation of services for distributed analysis at CMS J. Phys. Conf.Ser. 396(3) 032026[6] Riahi H et al. 2011 Large scale and low latency analysis facilities for the CMS experiment: development andoperational aspects J. Phys. Conf. Ser. 331 072030[7] The CI-Connect platform http://ci-connect.net[8] Jayatilaka B, Levshina T, Rynge M, Sehgal C and Slyz M 2015 The OSG open facility: A sharing ecosystemusing harvested opportunistic resources J. Phys. Conf. Ser. 664 032016[9] ATLAS Connect https://connect.usatlas.org[10] Duke Connect https://duke.ci-connect.net[11] Chard K, Lidman M, McCollam B, Bryan J, Ananthakrishnan R, Tuecke S and Foster I 2016 Globus Nexus: APlatform-as-a-Service provider of research identity, profile, and group management Future Gener. Comput.Syst. 56 57183[12] Blomer J, Aguado-Sa´nchez C, Buncic P and Harutyunyan A 2011 Distributing LHC application softwareand conditions databases using the CernVM file system J. Phys. Conf. Ser. 331 042003[13] Dorigo A, Elmer P, Furano F, and Hanushevsky A 2005 XROOTD - a highly scalable architecture for dataaccess. WSEAS Trans. Comp. 4(4) 34853[14] Dart E, Rotman L, Tierney B, Hester M and Zurawski J 2013 The Science DMZ: A network design patternfor data-intensive science SC: Proceedings of the International Conference on High Performance Computing,Networking, Storage and Analysis Denver, CO USA ACM 85:1-85:10[15] CMS Connect documentation http://docs.uscms.org/Selecting+Sites[16] Karavakis E, Andreeva J, Maier G and Khan A 2012 CMS dashboard for monitoring of the user analysisactivities, AIP Conf. Proc. 1504(1) 983-86[17] Jenkins https://jenkins.io

CMS Connect

https://digitalcommons.unl.edu/cgi/viewcontent.cgi?article=1252&amp;context=physicsfacpub

CMS Connect

Abstract

Similar works

Full text

Available Versions

DigitalCommons@University of Nebraska