A platform to deploy customized scientific virtual infrastructures on the cloud

This paper presents a software platform to dynamically deploy complex scientific virtual computing infrastructures, on top of Infrastructure as a Service (IaaS) Clouds. The platform orchestrates different services to provision the virtual computing resources. It dynamically installs the appropriate software to satisfy the requirements of a researcher, both on public and on-premise Clouds. The platform provides a web interface to enable the users to easily management of the lifecycle of virtual infrastructures. It enables users to define infrastructures, share them with other users, deploy and relinquish them, add or remove resources dynamically, create and share application recipes, etc. The paper also describes three case studies to deploy complex infrastructures, namely a Hadoop cluster, a single-node to perform NGS sequencing and a gateway for users to access the European Grid Infrastructure (EGI). This platform promotes a better use of on-premise hardware resources of a research center by allocating the computing resources just-in-time to the specific life time of the virtual infrastructures as well as the deployment of the very same infrastructures on a public Cloud.The authors would to thank the Spanish "Ministerio de Economia y Competitividad" for the project "Clusters Virtuales Elasticos y Migrables sobre Infraestructuras Cloud Hibridas" with reference TIN2013-44390-R.Caballer Fernández, M.; Segrelles Quilis, JD.; Moltó, G.; Blanquer Espert, I. (2015). A platform to deploy customized scientific virtual infrastructures on the cloud. Concurrency and Computation: Practice and Experience. 27(16):4318-4329. https://doi.org/10.1002/cpe.3518S431843292716Mell P Grance T The NIST definition of Cloud computing. NIST Special Publication 800-145 (Final) Technical Report 2011 http://csrc.nist.gov/publications/nistpubs/800-145/SP800-145.pdfBuyya, R., Broberg, J., & Goscinski, A. (Eds.). (2011). Cloud Computing. doi:10.1002/9780470940105Sahoo J Mohapatra S Lath R Virtualization: a survey on concepts, taxonomy and associated security issues 2010 Second International Conference on Computer and Network Technology Bangkok, Thailand 2010 222 226OpenStack OpenStack 2013 http://openstack.orgNurmi D Wolski R Grzegorczyk C Obertelli G Soman S Youseff L Zagorodnov D The Eucalyptus open-source Cloud-computing system Proceedings of 9th IEEE International Symposium on Cluster Computing and the Grid Shanghai, China 2009 124 131Amazon Web Services AWS CloudFormation http://aws.amazon.com/cloudformation/Amazon Web Services AWS OpsWorks http://aws.amazon.com/opsworks/Keahey K Freeman T Contextualization: providing one-click virtual clusters Fourth IEEE International Conference on eScience Indianapolis, Indiana, USA 2008 301 308Keahey K Freeman T Architecting a large-scale elastic environment: recontextualization and adaptive Cloud services for scientific computing 2012Marshall P Keahey K Freeman T Elastic site: using Clouds to elastically extend site resources Proceedings of the 2010 IEEE/ACM 10th International Conference on Cluster, Cloud and Grid Computing CCGRID '10 IEEE Computer Society, Washington, DC, USA 2010 43 52Bresnahan J Freeman T LaBissoniere D Keahey K Managing appliance launches in infrastructure Clouds Proceedings of the 2011 TeraGrid Conference: Extreme Digital Discovery TG '11 ACM, New York, NY, USA 2011 12:1 12:7Apache Whirr 2013 from:http://whirr.apache.org/Juve G Deelman E Automating application deployment in infrastructure clouds Proceedings of the 2011 IEEE Third International Conference on Cloud Computing Technology and Science CLOUDCOM '11 IEEE Computer Society, Washington, DC, USA 2011 658 665OASIS Topology and orchestration specification for cloud applications version 1.0 2013 http://docs.oasis-open.org/tosca/TOSCA/v1.0/TOSCA-v1.0.htmlBinz T Breitenbcher U Haupt F Kopp O Leymann F Nowak A Wagner S OpenTOSCA - a runtime for TOSCA-based cloud applications ICSOC, Lecture Notes in Computer Science 8274 Springer 2013 692 695Puppet Labs IT automation software for system administrators 2013 http://www.puppetlabs.com/Opscode Chef 2013 http://www.opscode.com/chef/DeHaan M Ansible 2013 http://ansible.cc/Vogels, W. (2008). Beyond server consolidation. Queue, 6(1), 20. doi:10.1145/1348583.1348590Carrión JV Moltó G De Alfonso C Caballer M Hernández V A generic catalog and repository service for virtual machine images 2nd International ICST Conference on Cloud Computing (CloudComp 2010) Barcelona, Spain 2010 1 15de Alfonso C Caballer M Alvarruiz F Molto G Hernández V Infrastructure deployment over the Cloud 2011 IEEE Third International Conference on Cloud Computing Technology and Science Athens, Greece 2011 517 521Caballer, M., Blanquer, I., Moltó, G., & de Alfonso, C. (2014). Dynamic Management of Virtual Infrastructures. Journal of Grid Computing, 13(1), 53-70. doi:10.1007/s10723-014-9296-5Dean, J., & Ghemawat, S. (2008). MapReduce. Communications of the ACM, 51(1), 107. doi:10.1145/1327452.1327492Shvachko K Kuang H Radia S Chansler R The Hadoop distributed file system 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST) Incline Village, NV, USA 2010 1 10Altschul, S. F., Gish, W., Miller, W., Myers, E. W., & Lipman, D. J. (1990). Basic local alignment search tool. Journal of Molecular Biology, 215(3), 403-410. doi:10.1016/s0022-2836(05)80360-

INDIGO-DataCloud: a Platform to Facilitate Seamless Access to E-Infrastructures

[EN] This paper describes the achievements of the H2020 project INDIGO-DataCloud. The project has provided e-infrastructures with tools, applications and cloud framework enhancements to manage the demanding requirements of scientific communities, either locally or through enhanced interfaces. The middleware developed allows to federate hybrid resources, to easily write, port and run scientific applications to the cloud. In particular, we have extended existing PaaS (Platform as a Service) solutions, allowing public and private e-infrastructures, including those provided by EGI, EUDAT, and Helix Nebula, to integrate their existing services and make them available through AAI services compliant with GEANT interfederation policies, thus guaranteeing transparency and trust in the provisioning of such services. Our middleware facilitates the execution of applications using containers on Cloud and Grid based infrastructures, as well as on HPC clusters. Our developments are freely downloadable as open source components, and are already being integrated into many scientific applications.INDIGO-Datacloud has been funded by the European Commision H2020 research and innovation program under grant agreement RIA 653549.Salomoni, D.; Campos, I.; Gaido, L.; Marco, J.; Solagna, P.; Gomes, J.; Matyska, L.... (2018). INDIGO-DataCloud: a Platform to Facilitate Seamless Access to E-Infrastructures. Journal of Grid Computing. 16(3):381-408. https://doi.org/10.1007/s10723-018-9453-3S381408163García, A.L., Castillo, E.F.-d., Puel, M.: Identity federation with VOMS in cloud infrastructures. In: 2013 IEEE 5Th International Conference on Cloud Computing Technology and Science, pp 42–48 (2013)Chadwick, D.W., Siu, K., Lee, C., Fouillat, Y., Germonville, D.: Adding federated identity management to OpenStack. Journal of Grid Computing 12(1), 3–27 (2014)Craig, A.L.: A design space review for general federation management using keystone. In: Proceedings of the 2014 IEEE/ACM 7th International Conference on Utility and Cloud Computing, pp 720–725. IEEE Computer Society (2014)Pustchi, N., Krishnan, R., Sandhu, R.: Authorization federation in iaas multi cloud. In: Proceedings of the 3rd International Workshop on Security in Cloud Computing, pp 63–71. ACM (2015)Lee, C.A., Desai, N., Brethorst, A.: A Keystone-Based Virtual Organization Management System. In: 2014 IEEE 6Th International Conference On Cloud Computing Technology and Science (Cloudcom), pp 727–730. IEEE (2014)Castillo, E.F.-d., Scardaci, D., García, A.L.: The EGI Federated Cloud e-Infrastructure. Procedia Computer Science 68, 196–205 (2015)AARC project: AARC Blueprint Architecture, see https://aarc-project.eu/architecture . Technical report (2016)Oesterle, F., Ostermann, S., Prodan, R., Mayr, G.J.: Experiences with distributed computing for meteorological applications: grid computing and cloud computing. Geosci. Model Dev. 8(7), 2067–2078 (2015)Plasencia, I.C., Castillo, E.F.-d., Heinemeyer, S., García, A.L., Pahlen, F., Borges, G.: Phenomenology tools on cloud infrastructures using OpenStack. The European Physical Journal C 73(4), 2375 (2013)Boettiger, C.: An introduction to docker for reproducible research. ACM SIGOPS Operating Systems Review 49(1), 71–79 (2015)Docker: http://www.docker.com (2013)Gomes, J., Campos, I., Bagnaschi, E., David, M., Alves, L., Martins, J., Pina, J., Alvaro, L.-G., Orviz, P.: Enabling rootless linux containers in multi-user environments: the udocker tool. Computing Physics Communications. https://doi.org/10.1016/j.cpc.2018.05.021 (2018)Zhang, Z., Chuan, W., Cheung, D.W.L.: A survey on cloud interoperability taxonomies, standards, and practice. SIGMETRICS perform. Eval. Rev. 40(4), 13–22 (2013)Lorido-Botran, T., Miguel-Alonso, J., Lozano, J.A.: A Review of Auto-scaling Techniques for Elastic Applications in Cloud Environments. Journal of Grid Computing 12(4), 559–592 (2014)Nyrén, R., Metsch, T., Edmonds, A., Papaspyrou, A.: Open Cloud Computing Interface–Core. Technical report, Open Grid Forum (2010)Metsch, T., Edmonds, A.: Open Cloud Computing Interface-Infrastructure. Technical report, Open Grid Forum (2010)Metsch, T., Edmonds, A.: Open Cloud Computing Interface-RESTful HTTP Rendering. Technical report, Open Grid Forum (2011)(Ca Technologies) Lipton, P., (Ibm) Moser, S., (Vnomic) Palma, D., (Ibm) Spatzier, T.: Topology and Orchestration Specification for Cloud Applications. Technical report, OASIS Standard (2013)Teckelmann, R., Reich, C., Sulistio, A.: Mapping of cloud standards to the taxonomy of interoperability in IaaS. In: Proceedings - 2011 3rd IEEE International Conference on Cloud Computing Technology and Science, CloudCom 2011, pp 522–526 (2011)García, A.L., Castillo, E.F.-d., Fernández, P.O.: Standards for enabling heterogeneous IaaS cloud federations. Computer Standards & Interfaces 47, 19–23 (2016)Caballer, M., Zala, S., García, A.L., Montó, G., Fernández, P.O., Velten, M.: Orchestrating complex application architectures in heterogeneous clouds. Journal of Grid Computing 16 (1), 3–18 (2018)Hardt, M., Jejkal, T., Plasencia, I.C., Castillo, E.F.-d., Jackson, A., Weiland, M., Palak, B., Plociennik, M., Nielsson, D.: Transparent Access to Scientific and Commercial Clouds from the Kepler Workflow Engine. Computing and Informatics 31(1), 119 (2012)Fakhfakh, F., Kacem, H.H., Kacem, A.H.: Workflow Scheduling in Cloud Computing a Survey. In: IEEE 18Th International Enterprise Distributed Object Computing Conference Workshops and Demonstrations (EDOCW), 2014, Vol. 71, pp. 372–378. Springer, New York (2014)Stockton, D.B., Santamaria, F.: Automating NEURON simulation deployment in cloud resources. Neuroinformatics 15(1), 51–70 (2017)Plóciennik, M., Fiore, S., Donvito, G., Owsiak, M., Fargetta, M., Barbera, R., Bruno, R., Giorgio, E., Williams, D.N., Aloisio, G.: Two-level Dynamic Workflow Orchestration in the INDIGO DataCloud for Large-scale, Climate Change Data Analytics Experiments. Procedia Computer Science 80, 722–733 (2016)Moreno-Vozmediano, R., Montero, R.S., Llorente, I.M.: Multicloud deployment of computing clusters for loosely coupled mtc applications. IEEE transactions on parallel and distributed systems 22(6), 924–930 (2011)Katsaros, G., Menzel, M., Lenk, A.: Cloud Service Orchestration with TOSCA, Chef and Openstack. In: Ic2e (2014)Garcia, A.L., Zangrando, L., Sgaravatto, M., Llorens, V., Vallero, S., Zaccolo, V., Bagnasco, S., Taneja, S., Dal Pra, S., Salomoni, D., Donvito, G.: Improved Cloud resource allocation: how INDIGO-DataCloud is overcoming the current limitations in Cloud schedulers. J. Phys. Conf. Ser. 898(9), 92010 (2017)Singh, S., Chana, I.: A survey on resource scheduling in cloud computing issues and challenges. Journal of Grid Computing, pp. 1–48 (2016)García, A.L., Castillo, E.F.-d., Fernández, P.O., Plasencia, I.C., de Lucas, J.M.: Resource provisioning in Science Clouds: Requirements and challenges. Software: Practice and Experience 48(3), 486–498 (2018)Chauhan, M.A., Babar, M.A., Benatallah, B.: Architecting cloud-enabled systems: a systematic survey of challenges and solutions. Software - Practice and Experience 47(4), 599–644 (2017)Somasundaram, T.S., Govindarajan, K.: CLOUDRB A Framework for scheduling and managing High-Performance Computing (HPC) applications in science cloud. Futur. Gener. Comput. Syst. 34, 47–65 (2014)Sotomayor, B., Keahey, K., Foster, I.: Overhead Matters: A Model for Virtual Resource Management. In: Proceedings of the 2nd International Workshop on Virtualization Technology in Distributed Computing SE - VTDC ’06, p 5. IEEE Computer Society, Washington (2006)SS, S.S., Shyam, G.K., Shyam, G.K.: Resource management for Infrastructure as a Service (IaaS) in cloud computing SS Manvi A survey. J. Netw. Comput. Appl. 41, 424–440 (2014)INDIGO-DataCloud consortium: Initial requirements from research communities - d2.1, see https://www.indigo-datacloud.eu/documents/initial-requirements-research-communities-d21 https://www.indigo-datacloud.eu/documents/initial-requirements-research-communities-d21 https://www.indigo-datacloud.eu/documents/initial-requirements-research-communities-d21 . Technical report (2015)Europen open science cloud: https://ec.europa.eu/research/openscience (2015)Proot: https://proot-me.github.io/ (2014)Runc: https://github.com/opencontainers/runc (2016)Fakechroot: https://github.com/dex4er/fakechroot (2015)Pérez, A., Moltó, G., Caballer, M., Calatrava, A.: Serverless computing for container-based architectures Future Generation Computer Systems (2018)de Vries, K.J.: Global fits of supersymmetric models after LHC run 1. Phd thesis Imperial College London (2015)Openstack: https://www.openstack.org/ (2015)See http://argus-documentation.readthedocs.io/en/stable/argus_introduction.html (2017)See https://en.wikipedia.org/wiki/xacml (2013)See http://www.simplecloud.info (2014)Opennebula: http://opennebula.org/ (2018)Redhat openshift: http://www.opencityplatform.eu (2011)The cloud foundry foundation: https://www.cloudfoundry.org/ (2015)Caballer, M., Blanquer, I., Moltó, G., de Alfonso, C.: Dynamic management of virtual infrastructures. Journal of Grid Computing 13(1), 53–70 (2015)See http://www.infoq.com/articles/scaling-docker-with-kubernetes http://www.infoq.com/articles/scaling-docker-with-kubernetes (2014)Prisma project: http://www.ponsmartcities-prisma.it/ (2010)Opencitiy platform: http://www.opencityplatform.eu (2014)Onedata: https://onedata.org/ (2018)Dynafed: http://lcgdm.web.cern.ch/dynafed-dynamic-federation-project http://lcgdm.web.cern.ch/dynafed-dynamic-federation-project (2011)Fts3: https://svnweb.cern.ch/trac/fts3 (2011)Fernández, P.O., García, A.L., Duma, D.C., Donvito, G., David, M., Gomes, J.: A set of common software quality assurance baseline criteria for research projects, see http://hdl.handle.net/10261/160086 . Technical reportHttermann, M.: Devops for developers Apress (2012)EOSC-Hub: ”Integrating and managing services for the European Open Science Cloud” Funded by H2020 research and innovation pr ogramme under grant agreement No. 777536. See http://eosc-hub.eu (2018)Apache License: author = https://www.apache.org/licenses/LICENSE-2.0 (2004)INDIGO Package Repo: http://repo.indigo-datacloud.eu/ (2017)INDIGO DockerHub: https://hub.docker.com/u/indigodatacloud/ https://hub.docker.com/u/indigodatacloud/ (2015)Indigo gitbook: https://indigo-dc.gitbooks.io/indigo-datacloud-releases https://indigo-dc.gitbooks.io/indigo-datacloud-releases (2017)Van Zundert, G.C., Bonvin, A.M.: Disvis: quantifying and visualizing the accessible interaction space of distance restrained biomolecular complexes. Bioinformatics 31(19), 3222–3224 (2015)Van Zundert, G.C., Bonvin, A.M.: Fast and sensitive rigid–body fitting into cryo–em density maps with powerfit. AIMS Biophys. 2(0273), 73–87 (2015

TOSCA-based orchestration of complex clusters at the IaaS level

[EN] This paper describes the adoption and extension of the TOSCA standard by the INDIGO-DataCloud project for the definition and deployment of complex computing clusters together with the required support in both OpenStack and OpenNebula, carried out in close collaboration with industry partners such as IBM. Two examples of these clusters are described in this paper, the definition of an elastic computing cluster to support the Galaxy bioinformatics application where the nodes are dynamically added and removed from the cluster to adapt to the workload, and the definition of an scalable Apache Mesos cluster for the execution of batch jobs and support for long-running services. The coupling of TOSCA with Ansible Roles to perform automated installation has resulted in the definition of high-level, deterministic templates to provision complex computing clusters across different Cloud sites.The authors would like to thank the European Commission for the financial support for project INDIGO-DataCloud (RIA 653549)Caballer Fernández, M.; Donvito, G.; Moltó, G.; Rocha, R.; Velten, M. (2017). TOSCA-based orchestration of complex clusters at the IaaS level. Journal of Physics: Conference Series (Online). 898:1-8. https://doi.org/10.1088/1742-6596/898/8/082036S1889

Dynamic management of virtual infrastructures

The final publication is available at Springer via http://dx.doi.org/10.1007/s10723-014-9296-5Cloud infrastructures are becoming an appropriate solution to address the computational needs of scientific applications. However, the use of public or on-premises Infrastructure as a Service (IaaS) clouds requires users to have non-trivial system administration skills. Resource provisioning systems provide facilities to choose the most suitable Virtual Machine Images (VMI) and basic configuration of multiple instances and subnetworks. Other tasks such as the configuration of cluster services, computational frameworks or specific applications are not trivial on the cloud, and normally users have to manually select the VMI that best fits, including undesired additional services and software packages. This paper presents a set of components that ease the access and the usability of IaaS clouds by automating the VMI selection, deployment, configuration, software installation, monitoring and update of Virtual Appliances. It supports APIs from a large number of virtual platforms, making user applications cloud-agnostic. In addition it integrates a contextualization system to enable the installation and configuration of all the user required applications providing the user with a fully functional infrastructure. Therefore, golden VMIs and configuration recipes can be easily reused across different deployments. Moreover, the contextualization agent included in the framework supports horizontal (increase/decrease the number of resources) and vertical (increase/decrease resources within a running Virtual Machine) by properly reconfiguring the software installed, considering the configuration of the multiple resources running. This paves the way for automatic virtual infrastructure deployment, customization and elastic modification at runtime for IaaS clouds.The authors would like to thank to thank the financial support received from the Ministerio de Economia y Competitividad for the project CodeCloud (TIN2010-17804).Caballer Fernández, M.; Blanquer Espert, I.; Moltó, G.; Alfonso Laguna, CD. (2015). Dynamic management of virtual infrastructures. Journal of Grid Computing. 13(1):53-70. https://doi.org/10.1007/s10723-014-9296-5S5370131de Alfonso, C., Caballer, M., Alvarruiz, F., Molto, G., Hernández, V.: Infrastructure deployment over the cloud. In: 2011 IEEE 3rd International Conference on Cloud Computing Technology and Science, pp. 517–521. IEEE. (2011). doi: 10.1109/CloudCom.2011.77Alvarruiz, F., De Alfonso, C., Caballer, M., Hernández, V.: An energy manager for high performance computer clusters. In: 2012 IEEE 10th International Symposium on Parallel and Distributed Processing with Applications, pp. 231–238. (2012). doi: 10.1109/ISPA.2012.38Amazon Web Services: AWS CloudFormation. (2013). http://aws.amazon.com/es/cloudformation/Apache: Whirr (2013). http://whirr.apache.org/Blanquer, I., Brasche, G., Lezzi, D.: Requirements of scientific applications in cloud offerings. In: Proceedings of the 2012 6th Iberian Grid Infrastructure Conference, IBERGRID ’12, pp. 173–182 (2012)Bresnahan, J., Freeman, T., LaBissoniere, D., Keahey, K.: Managing appliance launches in infrastructure clouds. In: Proceedings of the 2011 TeraGrid Conference: Extreme Digital Discovery, TG ’11, pp. 12:1–12:7. ACM, New York (2011). doi: 10.1145/2016741.2016755Buyya, R., Ranjan, R., Calheiros, R.N.: InterCloud: utility-oriented federation of cloud computing environments for scaling of application services. Algoritm. Archit. Parallel Process. 6081, 20 (2010)Buyya, R., Yeo, C.S., Venugopal, S., Broberg, J., Brandic, I.: Cloud computing and emerging IT platforms: Vision, hype, and reality for delivering computing as the 5th utility. Futur. Gener. Comput. Syst. 25(6), 599–616 (2009). doi: 10.1016/j.future.2008.12.001Caballer, M., De Alfonso, C., Alvarruiz, F., Moltó, G.: EC3: elastic cloud computing cluster. J. Comput. Syst. Sci. (2013). doi: 10.1016/j.jcss.2013.06.005Caballer, M., García, A., Moltó, G., de Alfonso, C.: Towards SLA-driven management of cloud infrastructures to elastically execute scientific applications. In: 6th Iberian Grid Infrastructure Conference (IberGrid), pp. 207–218 (2012)Carrión, J.V., Moltó, G., De Alfonso, C., Caballer, M., Hernández, V.: A generic catalog and repository service for virtual machine images. In: 2nd International ICST Conference on Cloud Computing CloudComp 2010 (2010)Cuomo, A., Modica, G., Distefano, S., Puliafito, A., Rak, M., Tomarchio, O., Venticinque, S., Villano, U.: An SLA-based broker for cloud infrastructures. J. Grid Comput 11(1), 1–25 (2012). doi: 10.1007/s10723-012-9241-4DeHaan, M.: Ansible. http://ansible.cc/ (2013)Distributed Management Task Force, Inc: Open Virtualization Format (OVF) (2010). http://dmtf.org/sites/default/files/standards/documents/DSP0243_1.1.0.pdfDistributed Management Task Force, Inc: Cloud Infrastructure Management Interface (CIMI) Model and REST Interface over HTTP Specification (2012). http://dmtf.org/sites/default/files/standards/documents/DSP0263_1.0.1.pdfEGI.eu: Seeking new horizons: EGI’s role for 2020. Tech. rep. (2012). https://documents.egi.eu/public/RetrieveFile?docid=1098&version=4&filename=EGI-1098-D230-final.pdfElmroth, E., Tordsson, J., Hernández, F.: Self-management challenges for multi-cloud architectures. Towards a service-based internet. Lect. Notes Comput. Sci. 6994, 38–49 (2011)HashiCorp: Vagrant (2013). http://www.vagrantup.com/Jacob, A.: Infrastructure in the cloud era. In: Proceedings of the 2009 International OReilly Conference Velocity (2009)Juve, G., Deelman, E.: Automating application deployment in infrastructure clouds. In: Proceedings of the 2011 IEEE 3rd International Conference on Cloud Computing Technology and Science, CLOUDCOM ’11, pp. 658–665. IEEE Computer Society, Washington DC (2011). doi: 10.1109/CloudCom.2011.102Keahey, K., Freeman, T.: Contextualization: providing one-click virtual clusters. In: 4th IEEE International Conference on eScience, pp. 301–308 (2008)Keahey, K., Freeman, T.: Architecting a large-scale elastic environment: recontextualization and adaptive cloud services for scientific computing (2012)Kecskemeti, G., Kertesz, A., Marosi, A., Kacsuk, P.: Interoperable resource management for establishing federated clouds. In: Achieving Federated and SelfManageable Cloud Infrastructures Theory and Practice, pp. 18–35 (2012). doi: 10.4018/978-1-4666-1631-8.ch002Kertesz, A., Kecskemeti, G., Oriol, M., Kotcauer, P., Acs, S., Rodríguez, M., Mercè, O., Marosi, A.C., Marco, J., Franch, X.: Enhancing federated cloud management with an integrated service monitoring approach. J. Grid Comput. 11(4), 699–720 (2013). doi: 10.1007/s10723-013-9269-0Loutas, N., Kamateri, E., Bosi, F., Tarabanis, K.: Cloud computing interoperability: the state of play. 2011 IEEE 3rd International Conference on Cloud Computing Technology and Science, pp. 752–757 (2011). doi: 10.1109/CloudCom.2011.116Marshall, P., Keahey, K., Freeman, T.: Elastic site: using clouds to elastically extend site resources. In: Proceedings of the 2010 IEEE/ACM 10th International Conference on Cluster, Cloud and Grid Computing, CCGRID ’10, pp. 43–52. IEEE Computer Society, Washington DC (2010). doi: 10.1109/CCGRID.2010.80Massie, M.L., Chun, B.N., Culler, D.E.: The ganglia distributed monitoring system: design, implementation, and experience. Parallel Comput. 30(5-6), 817–840 (2004)Mell, P., Grance, T.: The NIST definition of cloud computing. NIST Special Publication 800-145 (Final). Tech. rep. (2011). http://csrc.nist.gov/publications/nistpubs/800-145/SP800-145.pdfMoltó, G., Caballer, M., Romero, E., Alfonso, C.D.: Elastic memory management of virtualized infrastructures for applications with dynamic memory requirements. In: Proceedings of the International Conference on Computational Science ICCS 2013, pp. 159–168. Elsevier (2013). doi: 10.1016/j.procs.2013.05.179Morfeo: Claudia (2013). http://claudia.morfeo-project.org/wiki/index.php/Main_PageOASIS: Topology and Orchestration Specification for Cloud Applications Version 1.0 (2013). http://docs.oasis-open.org/tosca/TOSCA/v1.0/TOSCA-v1.0.htmlOCCI working group within the Open Grid Forum: Open Cloud Computing Interface Infrastructure (2011). http://ogf.org/documents/GFD.184.pdfOpscode: Chef (2013). http://www.opscode.com/chef/Pawluk, P., Simmons, B., Smit, M., Litoiu, M., Mankovski, S.: Introducing STRATOS: a cloud broker service. In: 2012 IEEE 5th International Conference on Cloud Computing, pp. 891–898 (2012). doi: 10.1109/CLOUD.2012.24Puppet Labs: IT Automation Software for System Administrators (2013). http://www.puppetlabs.com/Redl, C., Breskovic, I., Brandic, I., Dustdar, S.: Automatic SLA matching and provider selection in grid and cloud computing markets. In: Proceedings of the 2012 ACM/IEEE 13th International Conference on Grid Computing, GRID ’12, pp. 85–94. IEEE Computer Society, Washington (2012). doi: 10.1109/Grid.2012.18Rodero-Merino, L., Vaquero, L.M., Gil, V., Galán, F., Fontán, J., Montero, R.S., Llorente, I.M.: From infrastructure delivery to service management in clouds. Futur. Gener. Comput. Syst. 26(8), 1226–1240 (2010). doi: 10.1016/j.future.2010.02.013StratusLab: Claudia Platform (2013). http://stratuslab.eu/doku.php/claudiaSundareswaran, S., Squicciarini, A., Lin, D.: A brokerage-based approach for cloud service selection. In: Proceedings of the 2012 IEEE 5th International Conference on Cloud Computing, CLOUD ’12, pp. 558–565 (2012). doi: 10.1109/CLOUD.2012.119Telefónica Investigación y Desarrollo S.A. Unipersonal.: Telefónicas TCloud API Specification. (2010). http://www.tid.es/files/doc/apis/TCloud_API_Spec_v0.9.pdfYangui, S., Marshall, I.J., Laisne, J.P., Tata, S.: CompatibleOne: The open source cloud broker. J. Grid Comput. (2013). doi: 10.1007/s10723-013-9285-

A survey of the European Open Science Cloud services for expanding the capacity and capabilities of multidisciplinary scientific applications

Open Science is a paradigm in which scientific data, procedures, tools and results are shared transparently and reused by society. The European Open Science Cloud (EOSC) initiative is an effort in Europe to provide an open, trusted, virtual and federated computing environment to execute scientific applications and store, share and reuse research data across borders and scientific disciplines. Additionally, scientific services are becoming increasingly data-intensive, not only in terms of computationally intensive tasks but also in terms of storage resources. To meet those resource demands, computing paradigms such as High-Performance Computing (HPC) and Cloud Computing are applied to e-science applications. However, adapting applications and services to these paradigms is a challenging task, commonly requiring a deep knowledge of the underlying technologies, which often constitutes a general barrier to its uptake by scientists. In this context, EOSC-Synergy, a collaborative project involving more than 20 institutions from eight European countries pooling their knowledge and experience to enhance EOSC’s capabilities and capacities, aims to bring EOSC closer to the scientific communities. This article provides a summary analysis of the adaptations made in the ten thematic services of EOSC-Synergy to embrace this paradigm. These services are grouped into four categories: Earth Observation, Environment, Biomedicine, and Astrophysics. The analysis will lead to the identification of commonalities, best practices and common requirements, regardless of the thematic area of the service. Experience gained from the thematic services can be transferred to new services for the adoption of the EOSC ecosystem framework. The article made several recommendations for the integration of thematic services in the EOSC ecosystem regarding Authentication and Authorization (federated regional or thematic solutions based on EduGAIN mainly), FAIR data and metadata preservation solutions (both at cataloguing and data preservation—such as EUDAT’s B2SHARE), cloud platform-agnostic resource management services (such as Infrastructure Manager) and workload management solutions.This work was supported by the European Union’s Horizon 2020 research and innovation programme under grant agreement No 857647, EOSC-Synergy, European Open Science Cloud - Expanding Capacities by building Capabilities. Moreover, this work is partially funded by grant No 2015/24461-2, São Paulo Research Foundation (FAPESP). Francisco Brasileiro is a CNPq/Brazil researcher (grant 308027/2020-5).Peer Reviewed"Article signat per 20 autors/es: Amanda Calatrava, Hernán Asorey, Jan Astalos, Alberto Azevedo, Francesco Benincasa, Ignacio Blanquer, Martin Bobak, Francisco Brasileiro, Laia Codó, Laura del Cano, Borja Esteban, Meritxell Ferret, Josef Handl, Tobias Kerzenmacher, Valentin Kozlov, Aleš Křenek, Ricardo Martins, Manuel Pavesio, Antonio Juan Rubio-Montero, Juan Sánchez-Ferrero "Postprint (published version