Improved Cloud resource allocation: how INDIGO-Datacloud is overcoming the current limitations in Cloud schedulers

Trabajo presentado a: 22nd International Conference on Computing in High Energy and Nuclear Physics (CHEP2016) 10–14 October 2016, San Francisco.Performing efficient resource provisioning is a fundamental aspect for any resource provider. Local Resource Management Systems (LRMS) have been used in data centers for decades in order to obtain the best usage of the resources, providing their fair usage and partitioning for the users. In contrast, current cloud schedulers are normally based on the immediate allocation of resources on a first-come, first-served basis, meaning that a request will fail if there are no resources (e.g. OpenStack) or it will be trivially queued ordered by entry time (e.g. OpenNebula). Moreover, these scheduling strategies are based on a static partitioning of the resources, meaning that existing quotas cannot be exceeded, even if there are idle resources allocated to other projects. This is a consequence of the fact that cloud instances are not associated with a maximum execution time and leads to a situation where the resources are under-utilized. These facts have been identified by the INDIGO-DataCloud project as being too simplistic for accommodating scientific workloads in an efficient way, leading to an underutilization of the resources, a non desirable situation in scientific data centers. In this work, we will present the work done in the scheduling area during the first year of the INDIGO project and the foreseen evolutions.The authors want to acknowledge the support of the INDIGO-DataCloud (grant number 653549) project, funded by the European Commission’s Horizon 2020 Framework Programme.Peer Reviewe

The CloudVeneto initiative: 10 years of operations to support interdisciplinary open science

CloudVeneto is a private cloud targeted to scientific communities based on OpenStack software. It was designed in 2013 and put in operation one year later, to support INFN projects, mainly HEP ones. Its resources are physically distributed among two sites: the Physics Department of University of Padova-INFN Padova Unit and the INFN Legnaro National Laboratories. During these 10 years CloudVeneto evolved to integrate also resources funded by ten Departments of the University of Padova, and to support several scientific disciplines of different domains. The use cases the communities have to face up often show a common pattern. This was an opportunity for us to develop and improve the services on our infrastructure to provide common solutions to different use cases. It happened for example with the Container as a Service (CaaS) that makes the management of Kubernetes clusters easier from a user point of view. Moreover, CloudVeneto joined the INFN national cloud infrastructure (INFN Cloud), making available some resources to this federated infrastructure. CloudVeneto is also involved in an R&D project to realize a distributed analysis facility for the CMS experiment based on the HTCondor batch system. In this paper we describe some use-cases of different projects pointing out the common patterns and the new implementations and configurations done in the infrastructure

Implementation and use of a highly available and innovative IaaS solution: the Cloud Area Padovana

While in the business world the cloud paradigm is typically implemented purchasing resources and services from third party providers (e.g. Amazon), in the scientific environment there's usually the need of on-premises IaaS infrastructures which allow efficient usage of the hardware distributed among (and owned by) different scientific administrative domains. In addition, the requirement of open source adoption has led to the choice of products like OpenStack by many organizations. We describe a use case of the Italian National Institute for Nuclear Physics (INFN) which resulted in the implementation of a unique cloud service, called ’Cloud Area Padovana’, which encompasses resources spread over two different sites: the INFN Legnaro National Laboratories and the INFN Padova division. We describe how this IaaS has been implemented, which technologies have been adopted and how services have been configured in high-availability (HA) mode. We also discuss how identity and authorization management were implemented, adopting a widely accepted standard architecture based on SAML2 and OpenID: by leveraging the versatility of those standards the integration with authentication federations like IDEM was implemented. We also discuss some other innovative developments, such as a pluggable scheduler, implemented as an extension of the native OpenStack scheduler, which allows the allocation of resources according to a fair-share based model and which provides a persistent queuing mechanism for handling user requests that can not be immediately served. Tools, technologies, procedures used to install, configure, monitor, operate this cloud service are also discussed. Finally we present some examples that show how this IaaS infrastructure is being used

Search of the Orion spur for continuous gravitational waves using a loosely coherent algorithm on data from LIGO interferometers

We report results of a wideband search for periodic gravitational waves from isolated neutron stars within the Orion spur towards both the inner and outer regions of our Galaxy. As gravitational waves interact very weakly with matter, the search is unimpeded by dust and concentrations of stars. One search disk (A) is 6.87° in diameter and centered on 20h10m54.71s+33°33′25.29′′, and the other (B) is 7.45° in diameter and centered on 8h35m20.61s-46°49′25.151′′. We explored the frequency range of 50-1500 Hz and frequency derivative from 0 to -5×10-9 Hz/s. A multistage, loosely coherent search program allowed probing more deeply than before in these two regions, while increasing coherence length with every stage. Rigorous follow-up parameters have winnowed the initial coincidence set to only 70 candidates, to be examined manually. None of those 70 candidates proved to be consistent with an isolated gravitational-wave emitter, and 95% confidence level upper limits were placed on continuous-wave strain amplitudes. Near 169 Hz we achieve our lowest 95% C.L. upper limit on the worst-case linearly polarized strain amplitude h0 of 6.3×10-25, while at the high end of our frequency range we achieve a worst-case upper limit of 3.4×10-24 for all polarizations and sky locations. © 2016 American Physical Society

INDIGO-DataCloud: A data and computing platform to facilitate seamless access to e-infrastructures

This paper describes the achievements of the H2020 project INDIGO-DataCloud. The project has provided e-infrastructures with tools, applications and cloud framework enhancements to manage the demanding requirements of scientific communities, either locally or through enhanced interfaces. The middleware developed allows to federate hybrid resources, to easily write, port and run scientific applications to the cloud. In particular, we have extended existing PaaS (Platform as a Service) solutions, allowing public and private e-infrastructures, including those provided by EGI, EUDAT, and Helix Nebula, to integrate their existing services and make them available through AAI services compliant with GEANT interfederation policies, thus guaranteeing transparency and trust in the provisioning of such services. Our middleware facilitates the execution of applications using containers on Cloud and Grid based infrastructures, as well as on HPC clusters. Our developments are freely downloadable as open source components, and are already being integrated into many scientific applications

Search of the Orion spur for continuous gravitational waves using a loosely coherent algorithm on data from LIGO interferometers

We report results of a wideband search for periodic gravitational waves from isolated neutron stars within the Orion spur towards both the inner and outer regions of our Galaxy. As gravitational waves interact very weakly with matter, the search is unimpeded by dust and concentrations of stars. One search disk (A) is 6.87° in diameter and centered on 20[superscript h]10[superscript m]54.71[superscript s] + 33°33[superscript ′]25.29[superscript ′′], and the other (B) is 7.45° in diameter and centered on 8[superscript h]35[superscript m]20.61[superscript s] - 46°49[superscript ′]25.151[superscript ′′]. We explored the frequency range of 50–1500 Hz and frequency derivative from 0 to -5 × 10[superscript -9]  Hz/s. A multistage, loosely coherent search program allowed probing more deeply than before in these two regions, while increasing coherence length with every stage. Rigorous follow-up parameters have winnowed the initial coincidence set to only 70 candidates, to be examined manually. None of those 70 candidates proved to be consistent with an isolated gravitational-wave emitter, and 95% confidence level upper limits were placed on continuous-wave strain amplitudes. Near 169 Hz we achieve our lowest 95% C.L. upper limit on the worst-case linearly polarized strain amplitude h[subscript 0] of 6.3 × 10[superscript -25], while at the high end of our frequency range we achieve a worst-case upper limit of 3.4 × 10[superscript -24] for all polarizations and sky locations.National Science Foundation (U.S.)United States. National Aeronautics and Space AdministrationCarnegie TrustDavid & Lucile Packard FoundationAlfred P. Sloan Foundatio

First low frequency all-sky search for continuous gravitational wave signals

In this paper we present the results of the first low frequency all-sky search of continuous gravitational wave signals conducted on Virgo VSR2 and VSR4 data. The search covered the full sky, a frequency range between 20 and 128 Hz with a range of spin-down between −1.0×10−10 and +1.5×10−11  Hz/s, and was based on a hierarchical approach. The starting point was a set of short fast Fourier transforms, of length 8192 s, built from the calibrated strain data. Aggressive data cleaning, in both the time and frequency domains, has been done in order to remove, as much as possible, the effect of disturbances of instrumental origin. On each data set a number of candidates has been selected, using the FrequencyHough transform in an incoherent step. Only coincident candidates among VSR2 and VSR4 have been examined in order to strongly reduce the false alarm probability, and the most significant candidates have been selected. The criteria we have used for candidate selection and for the coincidence step greatly reduce the harmful effect of large instrumental artifacts. Selected candidates have been subject to a follow-up by constructing a new set of longer fast Fourier transforms followed by a further incoherent analysis, still based on the FrequencyHough transform. No evidence for continuous gravitational wave signals was found, and therefore we have set a population-based joint VSR2-VSR4 90% confidence level upper limit on the dimensionless gravitational wave strain in the frequency range between 20 and 128 Hz. This is the first all-sky search for continuous gravitational waves conducted, on data of ground-based interferometric detectors, at frequencies below 50 Hz. We set upper limits in the range between about 10−24 and 2×10−23 at most frequencies. Our upper limits on signal strain show an improvement of up to a factor of ∼2 with respect to the results of previous all-sky searches at frequencies below 80 H

All-sky search for long-duration gravitational wave transients with initial LIGO

We present the results of a search for long-duration gravitational wave transients in two sets of data collected by the LIGO Hanford and LIGO Livingston detectors between November 5, 2005 and September 30, 2007, and July 7, 2009 and October 20, 2010, with a total observational time of 283.0 days and 132.9 days, respectively. The search targets gravitational wave transients of duration 10–500 s in a frequency band of 40–1000 Hz, with minimal assumptions about the signal waveform, polarization, source direction, or time of occurrence. All candidate triggers were consistent with the expected background; as a result we set 90% confidence upper limits on the rate of long-duration gravitational wave transients for different types of gravitational wave signals. For signals from black hole accretion disk instabilities, we set upper limits on the source rate density between 3.4×10[superscript -5] and 9.4×10[superscript -4]  Mpc[superscript -3] yr[superscript -1] at 90% confidence. These are the first results from an all-sky search for unmodeled long-duration transient gravitational waves.Carnegie TrustDavid & Lucile Packard FoundationAlfred P. Sloan FoundationNational Science Foundation (U.S.

Search of the Orion spur for continuous gravitational waves using a loosely coherent algorithm on data from LIGO interferometers

We report results of a wideband search for periodic gravitational waves from isolated neutron stars within the Orion spur towards both the inner and outer regions of our Galaxy. As gravitational waves interact very weakly with matter, the search is unimpeded by dust and concentrations of stars. One search disk (A) is 6.87° in diameter and centered on 20[superscript h]10[superscript m]54.71[superscript s] + 33°33[superscript ′]25.29[superscript ′′], and the other (B) is 7.45° in diameter and centered on 8[superscript h]35[superscript m]20.61[superscript s] - 46°49[superscript ′]25.151[superscript ′′]. We explored the frequency range of 50–1500 Hz and frequency derivative from 0 to -5 × 10[superscript -9]  Hz/s. A multistage, loosely coherent search program allowed probing more deeply than before in these two regions, while increasing coherence length with every stage. Rigorous follow-up parameters have winnowed the initial coincidence set to only 70 candidates, to be examined manually. None of those 70 candidates proved to be consistent with an isolated gravitational-wave emitter, and 95% confidence level upper limits were placed on continuous-wave strain amplitudes. Near 169 Hz we achieve our lowest 95% C.L. upper limit on the worst-case linearly polarized strain amplitude h[subscript 0] of 6.3 × 10[superscript -25], while at the high end of our frequency range we achieve a worst-case upper limit of 3.4 × 10[superscript -24] for all polarizations and sky locations.National Science Foundation (U.S.)United States. National Aeronautics and Space AdministrationCarnegie TrustDavid & Lucile Packard FoundationAlfred P. Sloan Foundatio