INDIGO-DataCloud: a Platform to Facilitate Seamless Access to E-Infrastructures

[EN] This paper describes the achievements of the H2020 project INDIGO-DataCloud. The project has provided e-infrastructures with tools, applications and cloud framework enhancements to manage the demanding requirements of scientific communities, either locally or through enhanced interfaces. The middleware developed allows to federate hybrid resources, to easily write, port and run scientific applications to the cloud. In particular, we have extended existing PaaS (Platform as a Service) solutions, allowing public and private e-infrastructures, including those provided by EGI, EUDAT, and Helix Nebula, to integrate their existing services and make them available through AAI services compliant with GEANT interfederation policies, thus guaranteeing transparency and trust in the provisioning of such services. Our middleware facilitates the execution of applications using containers on Cloud and Grid based infrastructures, as well as on HPC clusters. Our developments are freely downloadable as open source components, and are already being integrated into many scientific applications.INDIGO-Datacloud has been funded by the European Commision H2020 research and innovation program under grant agreement RIA 653549.Salomoni, D.; Campos, I.; Gaido, L.; Marco, J.; Solagna, P.; Gomes, J.; Matyska, L.... (2018). INDIGO-DataCloud: a Platform to Facilitate Seamless Access to E-Infrastructures. Journal of Grid Computing. 16(3):381-408. https://doi.org/10.1007/s10723-018-9453-3S381408163García, A.L., Castillo, E.F.-d., Puel, M.: Identity federation with VOMS in cloud infrastructures. In: 2013 IEEE 5Th International Conference on Cloud Computing Technology and Science, pp 42–48 (2013)Chadwick, D.W., Siu, K., Lee, C., Fouillat, Y., Germonville, D.: Adding federated identity management to OpenStack. Journal of Grid Computing 12(1), 3–27 (2014)Craig, A.L.: A design space review for general federation management using keystone. In: Proceedings of the 2014 IEEE/ACM 7th International Conference on Utility and Cloud Computing, pp 720–725. IEEE Computer Society (2014)Pustchi, N., Krishnan, R., Sandhu, R.: Authorization federation in iaas multi cloud. In: Proceedings of the 3rd International Workshop on Security in Cloud Computing, pp 63–71. ACM (2015)Lee, C.A., Desai, N., Brethorst, A.: A Keystone-Based Virtual Organization Management System. In: 2014 IEEE 6Th International Conference On Cloud Computing Technology and Science (Cloudcom), pp 727–730. IEEE (2014)Castillo, E.F.-d., Scardaci, D., García, A.L.: The EGI Federated Cloud e-Infrastructure. Procedia Computer Science 68, 196–205 (2015)AARC project: AARC Blueprint Architecture, see https://aarc-project.eu/architecture . Technical report (2016)Oesterle, F., Ostermann, S., Prodan, R., Mayr, G.J.: Experiences with distributed computing for meteorological applications: grid computing and cloud computing. Geosci. Model Dev. 8(7), 2067–2078 (2015)Plasencia, I.C., Castillo, E.F.-d., Heinemeyer, S., García, A.L., Pahlen, F., Borges, G.: Phenomenology tools on cloud infrastructures using OpenStack. The European Physical Journal C 73(4), 2375 (2013)Boettiger, C.: An introduction to docker for reproducible research. ACM SIGOPS Operating Systems Review 49(1), 71–79 (2015)Docker: http://www.docker.com (2013)Gomes, J., Campos, I., Bagnaschi, E., David, M., Alves, L., Martins, J., Pina, J., Alvaro, L.-G., Orviz, P.: Enabling rootless linux containers in multi-user environments: the udocker tool. Computing Physics Communications. https://doi.org/10.1016/j.cpc.2018.05.021 (2018)Zhang, Z., Chuan, W., Cheung, D.W.L.: A survey on cloud interoperability taxonomies, standards, and practice. SIGMETRICS perform. Eval. Rev. 40(4), 13–22 (2013)Lorido-Botran, T., Miguel-Alonso, J., Lozano, J.A.: A Review of Auto-scaling Techniques for Elastic Applications in Cloud Environments. Journal of Grid Computing 12(4), 559–592 (2014)Nyrén, R., Metsch, T., Edmonds, A., Papaspyrou, A.: Open Cloud Computing Interface–Core. Technical report, Open Grid Forum (2010)Metsch, T., Edmonds, A.: Open Cloud Computing Interface-Infrastructure. Technical report, Open Grid Forum (2010)Metsch, T., Edmonds, A.: Open Cloud Computing Interface-RESTful HTTP Rendering. Technical report, Open Grid Forum (2011)(Ca Technologies) Lipton, P., (Ibm) Moser, S., (Vnomic) Palma, D., (Ibm) Spatzier, T.: Topology and Orchestration Specification for Cloud Applications. Technical report, OASIS Standard (2013)Teckelmann, R., Reich, C., Sulistio, A.: Mapping of cloud standards to the taxonomy of interoperability in IaaS. In: Proceedings - 2011 3rd IEEE International Conference on Cloud Computing Technology and Science, CloudCom 2011, pp 522–526 (2011)García, A.L., Castillo, E.F.-d., Fernández, P.O.: Standards for enabling heterogeneous IaaS cloud federations. Computer Standards & Interfaces 47, 19–23 (2016)Caballer, M., Zala, S., García, A.L., Montó, G., Fernández, P.O., Velten, M.: Orchestrating complex application architectures in heterogeneous clouds. Journal of Grid Computing 16 (1), 3–18 (2018)Hardt, M., Jejkal, T., Plasencia, I.C., Castillo, E.F.-d., Jackson, A., Weiland, M., Palak, B., Plociennik, M., Nielsson, D.: Transparent Access to Scientific and Commercial Clouds from the Kepler Workflow Engine. Computing and Informatics 31(1), 119 (2012)Fakhfakh, F., Kacem, H.H., Kacem, A.H.: Workflow Scheduling in Cloud Computing a Survey. In: IEEE 18Th International Enterprise Distributed Object Computing Conference Workshops and Demonstrations (EDOCW), 2014, Vol. 71, pp. 372–378. Springer, New York (2014)Stockton, D.B., Santamaria, F.: Automating NEURON simulation deployment in cloud resources. Neuroinformatics 15(1), 51–70 (2017)Plóciennik, M., Fiore, S., Donvito, G., Owsiak, M., Fargetta, M., Barbera, R., Bruno, R., Giorgio, E., Williams, D.N., Aloisio, G.: Two-level Dynamic Workflow Orchestration in the INDIGO DataCloud for Large-scale, Climate Change Data Analytics Experiments. Procedia Computer Science 80, 722–733 (2016)Moreno-Vozmediano, R., Montero, R.S., Llorente, I.M.: Multicloud deployment of computing clusters for loosely coupled mtc applications. IEEE transactions on parallel and distributed systems 22(6), 924–930 (2011)Katsaros, G., Menzel, M., Lenk, A.: Cloud Service Orchestration with TOSCA, Chef and Openstack. In: Ic2e (2014)Garcia, A.L., Zangrando, L., Sgaravatto, M., Llorens, V., Vallero, S., Zaccolo, V., Bagnasco, S., Taneja, S., Dal Pra, S., Salomoni, D., Donvito, G.: Improved Cloud resource allocation: how INDIGO-DataCloud is overcoming the current limitations in Cloud schedulers. J. Phys. Conf. Ser. 898(9), 92010 (2017)Singh, S., Chana, I.: A survey on resource scheduling in cloud computing issues and challenges. Journal of Grid Computing, pp. 1–48 (2016)García, A.L., Castillo, E.F.-d., Fernández, P.O., Plasencia, I.C., de Lucas, J.M.: Resource provisioning in Science Clouds: Requirements and challenges. Software: Practice and Experience 48(3), 486–498 (2018)Chauhan, M.A., Babar, M.A., Benatallah, B.: Architecting cloud-enabled systems: a systematic survey of challenges and solutions. Software - Practice and Experience 47(4), 599–644 (2017)Somasundaram, T.S., Govindarajan, K.: CLOUDRB A Framework for scheduling and managing High-Performance Computing (HPC) applications in science cloud. Futur. Gener. Comput. Syst. 34, 47–65 (2014)Sotomayor, B., Keahey, K., Foster, I.: Overhead Matters: A Model for Virtual Resource Management. In: Proceedings of the 2nd International Workshop on Virtualization Technology in Distributed Computing SE - VTDC ’06, p 5. IEEE Computer Society, Washington (2006)SS, S.S., Shyam, G.K., Shyam, G.K.: Resource management for Infrastructure as a Service (IaaS) in cloud computing SS Manvi A survey. J. Netw. Comput. Appl. 41, 424–440 (2014)INDIGO-DataCloud consortium: Initial requirements from research communities - d2.1, see https://www.indigo-datacloud.eu/documents/initial-requirements-research-communities-d21 https://www.indigo-datacloud.eu/documents/initial-requirements-research-communities-d21 https://www.indigo-datacloud.eu/documents/initial-requirements-research-communities-d21 . Technical report (2015)Europen open science cloud: https://ec.europa.eu/research/openscience (2015)Proot: https://proot-me.github.io/ (2014)Runc: https://github.com/opencontainers/runc (2016)Fakechroot: https://github.com/dex4er/fakechroot (2015)Pérez, A., Moltó, G., Caballer, M., Calatrava, A.: Serverless computing for container-based architectures Future Generation Computer Systems (2018)de Vries, K.J.: Global fits of supersymmetric models after LHC run 1. Phd thesis Imperial College London (2015)Openstack: https://www.openstack.org/ (2015)See http://argus-documentation.readthedocs.io/en/stable/argus_introduction.html (2017)See https://en.wikipedia.org/wiki/xacml (2013)See http://www.simplecloud.info (2014)Opennebula: http://opennebula.org/ (2018)Redhat openshift: http://www.opencityplatform.eu (2011)The cloud foundry foundation: https://www.cloudfoundry.org/ (2015)Caballer, M., Blanquer, I., Moltó, G., de Alfonso, C.: Dynamic management of virtual infrastructures. Journal of Grid Computing 13(1), 53–70 (2015)See http://www.infoq.com/articles/scaling-docker-with-kubernetes http://www.infoq.com/articles/scaling-docker-with-kubernetes (2014)Prisma project: http://www.ponsmartcities-prisma.it/ (2010)Opencitiy platform: http://www.opencityplatform.eu (2014)Onedata: https://onedata.org/ (2018)Dynafed: http://lcgdm.web.cern.ch/dynafed-dynamic-federation-project http://lcgdm.web.cern.ch/dynafed-dynamic-federation-project (2011)Fts3: https://svnweb.cern.ch/trac/fts3 (2011)Fernández, P.O., García, A.L., Duma, D.C., Donvito, G., David, M., Gomes, J.: A set of common software quality assurance baseline criteria for research projects, see http://hdl.handle.net/10261/160086 . Technical reportHttermann, M.: Devops for developers Apress (2012)EOSC-Hub: ”Integrating and managing services for the European Open Science Cloud” Funded by H2020 research and innovation pr ogramme under grant agreement No. 777536. See http://eosc-hub.eu (2018)Apache License: author = https://www.apache.org/licenses/LICENSE-2.0 (2004)INDIGO Package Repo: http://repo.indigo-datacloud.eu/ (2017)INDIGO DockerHub: https://hub.docker.com/u/indigodatacloud/ https://hub.docker.com/u/indigodatacloud/ (2015)Indigo gitbook: https://indigo-dc.gitbooks.io/indigo-datacloud-releases https://indigo-dc.gitbooks.io/indigo-datacloud-releases (2017)Van Zundert, G.C., Bonvin, A.M.: Disvis: quantifying and visualizing the accessible interaction space of distance restrained biomolecular complexes. Bioinformatics 31(19), 3222–3224 (2015)Van Zundert, G.C., Bonvin, A.M.: Fast and sensitive rigid–body fitting into cryo–em density maps with powerfit. AIMS Biophys. 2(0273), 73–87 (2015

Insights from Learning Analytics for Hands-On Cloud Computing Labs in AWS

[EN] Cloud computing instruction requires hands-on experience with a myriad of distributed computing services from a public cloud provider. Tracking the progress of the students, especially for online courses, requires one to automatically gather evidence and produce learning analytics in order to further determine the behavior and performance of students. With this aim, this paper describes the experience from an online course in cloud computing with Amazon Web Services on the creation of an open-source data processing tool to systematically obtain learning analytics related to the hands-on activities carried out throughout the course. These data, combined with the data obtained from the learning management system, have allowed the better characterization of the behavior of students in the course. Insights from a population of more than 420 online students through three academic years have been assessed, the dataset has been released for increased reproducibility. The results corroborate that course length has an impact on online students dropout. In addition, a gender analysis pointed out that there are no statistically significant differences in the final marks between genders, but women show an increased degree of commitment with the activities planned in the course.This research was funded by the Spanish "Ministerio de Economia, Industria y Competitividad through grant number TIN2016-79951-R (BigCLOE)", the "Vicerrectorado de Estudios, Calidad y Acreditacion" of the Universitat Politecnica de Valencia (UPV) to develop the PIME B29 and PIME/19-20/166, and by the Conselleria d'Innovacio, Universitat, Ciencia i Societat Digital for the project "CloudSTEM" with reference number AICO/2019/313.Moltó, G.; Naranjo-Delgado, DM.; Segrelles Quilis, JD. (2020). Insights from Learning Analytics for Hands-On Cloud Computing Labs in AWS. Applied Sciences. 10(24):1-13. https://doi.org/10.3390/app10249148S1131024Motiwalla, L., Deokar, A. V., Sarnikar, S., & Dimoka, A. (2019). Leveraging Data Analytics for Behavioral Research. Information Systems Frontiers, 21(4), 735-742. doi:10.1007/s10796-019-09928-8Siemens, G., & Baker, R. S. J. d. (2012). Learning analytics and educational data mining. Proceedings of the 2nd International Conference on Learning Analytics and Knowledge - LAK ’12. doi:10.1145/2330601.2330661Blikstein, P. (2013). Multimodal learning analytics. Proceedings of the Third International Conference on Learning Analytics and Knowledge - LAK ’13. doi:10.1145/2460296.2460316Hewson, E. R. F. (2018). Students’ Emotional Engagement, Motivation and Behaviour Over the Life of an Online Course: Reflections on Two Market Research Case Studies. Journal of Interactive Media in Education, 2018(1). doi:10.5334/jime.472Kahan, T., Soffer, T., & Nachmias, R. (2017). Types of Participant Behavior in a Massive Open Online Course. The International Review of Research in Open and Distributed Learning, 18(6). doi:10.19173/irrodl.v18i6.3087Cross, S., & Whitelock, D. (2016). Similarity and difference in fee-paying and no-fee learner expectations, interaction and reaction to learning in a massive open online course. Interactive Learning Environments, 25(4), 439-451. doi:10.1080/10494820.2016.1138312Charleer, S., Klerkx, J., & Duval, E. (2014). Learning Dashboards. Journal of Learning Analytics, 1(3), 199-202. doi:10.18608/jla.2014.13.22Worsley, M. (2012). Multimodal learning analytics. Proceedings of the 14th ACM international conference on Multimodal interaction - ICMI ’12. doi:10.1145/2388676.2388755Spikol, D., Prieto, L. P., Rodríguez-Triana, M. J., Worsley, M., Ochoa, X., Cukurova, M., … Ringtved, U. L. (2017). Current and future multimodal learning analytics data challenges. Proceedings of the Seventh International Learning Analytics & Knowledge Conference. doi:10.1145/3027385.3029437Ochoa, X., Worsley, M., Weibel, N., & Oviatt, S. (2016). Multimodal learning analytics data challenges. Proceedings of the Sixth International Conference on Learning Analytics & Knowledge - LAK ’16. doi:10.1145/2883851.2883913Aguilar, J., Sánchez, M., Cordero, J., Valdiviezo-Díaz, P., Barba-Guamán, L., & Chamba-Eras, L. (2017). Learning analytics tasks as services in smart classrooms. Universal Access in the Information Society, 17(4), 693-709. doi:10.1007/s10209-017-0525-0Lu, O. H. T., Huang, J. C. H., Huang, A. Y. Q., & Yang, S. J. H. (2017). Applying learning analytics for improving students engagement and learning outcomes in an MOOCs enabled collaborative programming course. Interactive Learning Environments, 25(2), 220-234. doi:10.1080/10494820.2016.1278391Drachsler, H., & Kalz, M. (2016). The MOOC and learning analytics innovation cycle (MOLAC): a reflective summary of ongoing research and its challenges. Journal of Computer Assisted Learning, 32(3), 281-290. doi:10.1111/jcal.12135Ruiperez-Valiente, J. A., Munoz-Merino, P. J., Gascon-Pinedo, J. A., & Kloos, C. D. (2017). Scaling to Massiveness With ANALYSE: A Learning Analytics Tool for Open edX. IEEE Transactions on Human-Machine Systems, 47(6), 909-914. doi:10.1109/thms.2016.2630420Er, E., Gómez-Sánchez, E., Dimitriadis, Y., Bote-Lorenzo, M. L., Asensio-Pérez, J. I., & Álvarez-Álvarez, S. (2019). Aligning learning design and learning analytics through instructor involvement: a MOOC case study. Interactive Learning Environments, 27(5-6), 685-698. doi:10.1080/10494820.2019.1610455Tabaa, Y., & Medouri, A. (2013). LASyM: A Learning Analytics System for MOOCs. International Journal of Advanced Computer Science and Applications, 4(5). doi:10.14569/ijacsa.2013.040516Shorfuzzaman, M., Hossain, M. S., Nazir, A., Muhammad, G., & Alamri, A. (2019). Harnessing the power of big data analytics in the cloud to support learning analytics in mobile learning environment. Computers in Human Behavior, 92, 578-588. doi:10.1016/j.chb.2018.07.002Klašnja-Milićević, A., Ivanović, M., & Budimac, Z. (2017). Data science in education: Big data and learning analytics. Computer Applications in Engineering Education, 25(6), 1066-1078. doi:10.1002/cae.21844Logglyhttps://www.loggly.com/Molto, G., & Caballer, M. (2014). On using the cloud to support online courses. 2014 IEEE Frontiers in Education Conference (FIE) Proceedings. doi:10.1109/fie.2014.7044041Caballer, M., Blanquer, I., Moltó, G., & de Alfonso, C. (2014). Dynamic Management of Virtual Infrastructures. Journal of Grid Computing, 13(1), 53-70. doi:10.1007/s10723-014-9296-5AWS CloudTrailhttps://aws.amazon.com/cloudtrail/Amazon Simple Storage Service (Amazon S3)http://aws.amazon.com/s3/Naranjo, D. M., Prieto, J. R., Moltó, G., & Calatrava, A. (2019). A Visual Dashboard to Track Learning Analytics for Educational Cloud Computing. Sensors, 19(13), 2952. doi:10.3390/s19132952Baldini, I., Castro, P., Chang, K., Cheng, P., Fink, S., Ishakian, V., … Suter, P. (2017). Serverless Computing: Current Trends and Open Problems. Research Advances in Cloud Computing, 1-20. doi:10.1007/978-981-10-5026-8_1Zimmerman, D. W. (1987). Comparative Power of StudentTTest and Mann-WhitneyUTest for Unequal Sample Sizes and Variances. The Journal of Experimental Education, 55(3), 171-174. doi:10.1080/00220973.1987.10806451Kruskal, W. H., & Wallis, W. A. (1952). Use of Ranks in One-Criterion Variance Analysis. Journal of the American Statistical Association, 47(260), 583-621. doi:10.1080/01621459.1952.10483441Voyer, D., & Voyer, S. D. (2014). Gender differences in scholastic achievement: A meta-analysis. Psychological Bulletin, 140(4), 1174-1204. doi:10.1037/a0036620Ellemers, N., Heuvel, H., Gilder, D., Maass, A., & Bonvini, A. (2004). The underrepresentation of women in science: Differential commitment or the queen bee syndrome? British Journal of Social Psychology, 43(3), 315-338. doi:10.1348/0144666042037999Sheard, M. (2009). Hardiness commitment, gender, and age differentiate university academic performance. British Journal of Educational Psychology, 79(1), 189-204. doi:10.1348/000709908x30440

A platform to deploy customized scientific virtual infrastructures on the cloud

This paper presents a software platform to dynamically deploy complex scientific virtual computing infrastructures, on top of Infrastructure as a Service (IaaS) Clouds. The platform orchestrates different services to provision the virtual computing resources. It dynamically installs the appropriate software to satisfy the requirements of a researcher, both on public and on-premise Clouds. The platform provides a web interface to enable the users to easily management of the lifecycle of virtual infrastructures. It enables users to define infrastructures, share them with other users, deploy and relinquish them, add or remove resources dynamically, create and share application recipes, etc. The paper also describes three case studies to deploy complex infrastructures, namely a Hadoop cluster, a single-node to perform NGS sequencing and a gateway for users to access the European Grid Infrastructure (EGI). This platform promotes a better use of on-premise hardware resources of a research center by allocating the computing resources just-in-time to the specific life time of the virtual infrastructures as well as the deployment of the very same infrastructures on a public Cloud.The authors would to thank the Spanish "Ministerio de Economia y Competitividad" for the project "Clusters Virtuales Elasticos y Migrables sobre Infraestructuras Cloud Hibridas" with reference TIN2013-44390-R.Caballer Fernández, M.; Segrelles Quilis, JD.; Moltó, G.; Blanquer Espert, I. (2015). A platform to deploy customized scientific virtual infrastructures on the cloud. Concurrency and Computation: Practice and Experience. 27(16):4318-4329. https://doi.org/10.1002/cpe.3518S431843292716Mell P Grance T The NIST definition of Cloud computing. NIST Special Publication 800-145 (Final) Technical Report 2011 http://csrc.nist.gov/publications/nistpubs/800-145/SP800-145.pdfBuyya, R., Broberg, J., & Goscinski, A. (Eds.). (2011). Cloud Computing. doi:10.1002/9780470940105Sahoo J Mohapatra S Lath R Virtualization: a survey on concepts, taxonomy and associated security issues 2010 Second International Conference on Computer and Network Technology Bangkok, Thailand 2010 222 226OpenStack OpenStack 2013 http://openstack.orgNurmi D Wolski R Grzegorczyk C Obertelli G Soman S Youseff L Zagorodnov D The Eucalyptus open-source Cloud-computing system Proceedings of 9th IEEE International Symposium on Cluster Computing and the Grid Shanghai, China 2009 124 131Amazon Web Services AWS CloudFormation http://aws.amazon.com/cloudformation/Amazon Web Services AWS OpsWorks http://aws.amazon.com/opsworks/Keahey K Freeman T Contextualization: providing one-click virtual clusters Fourth IEEE International Conference on eScience Indianapolis, Indiana, USA 2008 301 308Keahey K Freeman T Architecting a large-scale elastic environment: recontextualization and adaptive Cloud services for scientific computing 2012Marshall P Keahey K Freeman T Elastic site: using Clouds to elastically extend site resources Proceedings of the 2010 IEEE/ACM 10th International Conference on Cluster, Cloud and Grid Computing CCGRID '10 IEEE Computer Society, Washington, DC, USA 2010 43 52Bresnahan J Freeman T LaBissoniere D Keahey K Managing appliance launches in infrastructure Clouds Proceedings of the 2011 TeraGrid Conference: Extreme Digital Discovery TG '11 ACM, New York, NY, USA 2011 12:1 12:7Apache Whirr 2013 from:http://whirr.apache.org/Juve G Deelman E Automating application deployment in infrastructure clouds Proceedings of the 2011 IEEE Third International Conference on Cloud Computing Technology and Science CLOUDCOM '11 IEEE Computer Society, Washington, DC, USA 2011 658 665OASIS Topology and orchestration specification for cloud applications version 1.0 2013 http://docs.oasis-open.org/tosca/TOSCA/v1.0/TOSCA-v1.0.htmlBinz T Breitenbcher U Haupt F Kopp O Leymann F Nowak A Wagner S OpenTOSCA - a runtime for TOSCA-based cloud applications ICSOC, Lecture Notes in Computer Science 8274 Springer 2013 692 695Puppet Labs IT automation software for system administrators 2013 http://www.puppetlabs.com/Opscode Chef 2013 http://www.opscode.com/chef/DeHaan M Ansible 2013 http://ansible.cc/Vogels, W. (2008). Beyond server consolidation. Queue, 6(1), 20. doi:10.1145/1348583.1348590Carrión JV Moltó G De Alfonso C Caballer M Hernández V A generic catalog and repository service for virtual machine images 2nd International ICST Conference on Cloud Computing (CloudComp 2010) Barcelona, Spain 2010 1 15de Alfonso C Caballer M Alvarruiz F Molto G Hernández V Infrastructure deployment over the Cloud 2011 IEEE Third International Conference on Cloud Computing Technology and Science Athens, Greece 2011 517 521Caballer, M., Blanquer, I., Moltó, G., & de Alfonso, C. (2014). Dynamic Management of Virtual Infrastructures. Journal of Grid Computing, 13(1), 53-70. doi:10.1007/s10723-014-9296-5Dean, J., & Ghemawat, S. (2008). MapReduce. Communications of the ACM, 51(1), 107. doi:10.1145/1327452.1327492Shvachko K Kuang H Radia S Chansler R The Hadoop distributed file system 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST) Incline Village, NV, USA 2010 1 10Altschul, S. F., Gish, W., Miller, W., Myers, E. W., & Lipman, D. J. (1990). Basic local alignment search tool. Journal of Molecular Biology, 215(3), 403-410. doi:10.1016/s0022-2836(05)80360-

Dynamic management of virtual infrastructures

The final publication is available at Springer via http://dx.doi.org/10.1007/s10723-014-9296-5Cloud infrastructures are becoming an appropriate solution to address the computational needs of scientific applications. However, the use of public or on-premises Infrastructure as a Service (IaaS) clouds requires users to have non-trivial system administration skills. Resource provisioning systems provide facilities to choose the most suitable Virtual Machine Images (VMI) and basic configuration of multiple instances and subnetworks. Other tasks such as the configuration of cluster services, computational frameworks or specific applications are not trivial on the cloud, and normally users have to manually select the VMI that best fits, including undesired additional services and software packages. This paper presents a set of components that ease the access and the usability of IaaS clouds by automating the VMI selection, deployment, configuration, software installation, monitoring and update of Virtual Appliances. It supports APIs from a large number of virtual platforms, making user applications cloud-agnostic. In addition it integrates a contextualization system to enable the installation and configuration of all the user required applications providing the user with a fully functional infrastructure. Therefore, golden VMIs and configuration recipes can be easily reused across different deployments. Moreover, the contextualization agent included in the framework supports horizontal (increase/decrease the number of resources) and vertical (increase/decrease resources within a running Virtual Machine) by properly reconfiguring the software installed, considering the configuration of the multiple resources running. This paves the way for automatic virtual infrastructure deployment, customization and elastic modification at runtime for IaaS clouds.The authors would like to thank to thank the financial support received from the Ministerio de Economia y Competitividad for the project CodeCloud (TIN2010-17804).Caballer Fernández, M.; Blanquer Espert, I.; Moltó, G.; Alfonso Laguna, CD. (2015). Dynamic management of virtual infrastructures. Journal of Grid Computing. 13(1):53-70. https://doi.org/10.1007/s10723-014-9296-5S5370131de Alfonso, C., Caballer, M., Alvarruiz, F., Molto, G., Hernández, V.: Infrastructure deployment over the cloud. In: 2011 IEEE 3rd International Conference on Cloud Computing Technology and Science, pp. 517–521. IEEE. (2011). doi: 10.1109/CloudCom.2011.77Alvarruiz, F., De Alfonso, C., Caballer, M., Hernández, V.: An energy manager for high performance computer clusters. In: 2012 IEEE 10th International Symposium on Parallel and Distributed Processing with Applications, pp. 231–238. (2012). doi: 10.1109/ISPA.2012.38Amazon Web Services: AWS CloudFormation. (2013). http://aws.amazon.com/es/cloudformation/Apache: Whirr (2013). http://whirr.apache.org/Blanquer, I., Brasche, G., Lezzi, D.: Requirements of scientific applications in cloud offerings. In: Proceedings of the 2012 6th Iberian Grid Infrastructure Conference, IBERGRID ’12, pp. 173–182 (2012)Bresnahan, J., Freeman, T., LaBissoniere, D., Keahey, K.: Managing appliance launches in infrastructure clouds. In: Proceedings of the 2011 TeraGrid Conference: Extreme Digital Discovery, TG ’11, pp. 12:1–12:7. ACM, New York (2011). doi: 10.1145/2016741.2016755Buyya, R., Ranjan, R., Calheiros, R.N.: InterCloud: utility-oriented federation of cloud computing environments for scaling of application services. Algoritm. Archit. Parallel Process. 6081, 20 (2010)Buyya, R., Yeo, C.S., Venugopal, S., Broberg, J., Brandic, I.: Cloud computing and emerging IT platforms: Vision, hype, and reality for delivering computing as the 5th utility. Futur. Gener. Comput. Syst. 25(6), 599–616 (2009). doi: 10.1016/j.future.2008.12.001Caballer, M., De Alfonso, C., Alvarruiz, F., Moltó, G.: EC3: elastic cloud computing cluster. J. Comput. Syst. Sci. (2013). doi: 10.1016/j.jcss.2013.06.005Caballer, M., García, A., Moltó, G., de Alfonso, C.: Towards SLA-driven management of cloud infrastructures to elastically execute scientific applications. In: 6th Iberian Grid Infrastructure Conference (IberGrid), pp. 207–218 (2012)Carrión, J.V., Moltó, G., De Alfonso, C., Caballer, M., Hernández, V.: A generic catalog and repository service for virtual machine images. In: 2nd International ICST Conference on Cloud Computing CloudComp 2010 (2010)Cuomo, A., Modica, G., Distefano, S., Puliafito, A., Rak, M., Tomarchio, O., Venticinque, S., Villano, U.: An SLA-based broker for cloud infrastructures. J. Grid Comput 11(1), 1–25 (2012). doi: 10.1007/s10723-012-9241-4DeHaan, M.: Ansible. http://ansible.cc/ (2013)Distributed Management Task Force, Inc: Open Virtualization Format (OVF) (2010). http://dmtf.org/sites/default/files/standards/documents/DSP0243_1.1.0.pdfDistributed Management Task Force, Inc: Cloud Infrastructure Management Interface (CIMI) Model and REST Interface over HTTP Specification (2012). http://dmtf.org/sites/default/files/standards/documents/DSP0263_1.0.1.pdfEGI.eu: Seeking new horizons: EGI’s role for 2020. Tech. rep. (2012). https://documents.egi.eu/public/RetrieveFile?docid=1098&version=4&filename=EGI-1098-D230-final.pdfElmroth, E., Tordsson, J., Hernández, F.: Self-management challenges for multi-cloud architectures. Towards a service-based internet. Lect. Notes Comput. Sci. 6994, 38–49 (2011)HashiCorp: Vagrant (2013). http://www.vagrantup.com/Jacob, A.: Infrastructure in the cloud era. In: Proceedings of the 2009 International OReilly Conference Velocity (2009)Juve, G., Deelman, E.: Automating application deployment in infrastructure clouds. In: Proceedings of the 2011 IEEE 3rd International Conference on Cloud Computing Technology and Science, CLOUDCOM ’11, pp. 658–665. IEEE Computer Society, Washington DC (2011). doi: 10.1109/CloudCom.2011.102Keahey, K., Freeman, T.: Contextualization: providing one-click virtual clusters. In: 4th IEEE International Conference on eScience, pp. 301–308 (2008)Keahey, K., Freeman, T.: Architecting a large-scale elastic environment: recontextualization and adaptive cloud services for scientific computing (2012)Kecskemeti, G., Kertesz, A., Marosi, A., Kacsuk, P.: Interoperable resource management for establishing federated clouds. In: Achieving Federated and SelfManageable Cloud Infrastructures Theory and Practice, pp. 18–35 (2012). doi: 10.4018/978-1-4666-1631-8.ch002Kertesz, A., Kecskemeti, G., Oriol, M., Kotcauer, P., Acs, S., Rodríguez, M., Mercè, O., Marosi, A.C., Marco, J., Franch, X.: Enhancing federated cloud management with an integrated service monitoring approach. J. Grid Comput. 11(4), 699–720 (2013). doi: 10.1007/s10723-013-9269-0Loutas, N., Kamateri, E., Bosi, F., Tarabanis, K.: Cloud computing interoperability: the state of play. 2011 IEEE 3rd International Conference on Cloud Computing Technology and Science, pp. 752–757 (2011). doi: 10.1109/CloudCom.2011.116Marshall, P., Keahey, K., Freeman, T.: Elastic site: using clouds to elastically extend site resources. In: Proceedings of the 2010 IEEE/ACM 10th International Conference on Cluster, Cloud and Grid Computing, CCGRID ’10, pp. 43–52. IEEE Computer Society, Washington DC (2010). doi: 10.1109/CCGRID.2010.80Massie, M.L., Chun, B.N., Culler, D.E.: The ganglia distributed monitoring system: design, implementation, and experience. Parallel Comput. 30(5-6), 817–840 (2004)Mell, P., Grance, T.: The NIST definition of cloud computing. NIST Special Publication 800-145 (Final). Tech. rep. (2011). http://csrc.nist.gov/publications/nistpubs/800-145/SP800-145.pdfMoltó, G., Caballer, M., Romero, E., Alfonso, C.D.: Elastic memory management of virtualized infrastructures for applications with dynamic memory requirements. In: Proceedings of the International Conference on Computational Science ICCS 2013, pp. 159–168. Elsevier (2013). doi: 10.1016/j.procs.2013.05.179Morfeo: Claudia (2013). http://claudia.morfeo-project.org/wiki/index.php/Main_PageOASIS: Topology and Orchestration Specification for Cloud Applications Version 1.0 (2013). http://docs.oasis-open.org/tosca/TOSCA/v1.0/TOSCA-v1.0.htmlOCCI working group within the Open Grid Forum: Open Cloud Computing Interface Infrastructure (2011). http://ogf.org/documents/GFD.184.pdfOpscode: Chef (2013). http://www.opscode.com/chef/Pawluk, P., Simmons, B., Smit, M., Litoiu, M., Mankovski, S.: Introducing STRATOS: a cloud broker service. In: 2012 IEEE 5th International Conference on Cloud Computing, pp. 891–898 (2012). doi: 10.1109/CLOUD.2012.24Puppet Labs: IT Automation Software for System Administrators (2013). http://www.puppetlabs.com/Redl, C., Breskovic, I., Brandic, I., Dustdar, S.: Automatic SLA matching and provider selection in grid and cloud computing markets. In: Proceedings of the 2012 ACM/IEEE 13th International Conference on Grid Computing, GRID ’12, pp. 85–94. IEEE Computer Society, Washington (2012). doi: 10.1109/Grid.2012.18Rodero-Merino, L., Vaquero, L.M., Gil, V., Galán, F., Fontán, J., Montero, R.S., Llorente, I.M.: From infrastructure delivery to service management in clouds. Futur. Gener. Comput. Syst. 26(8), 1226–1240 (2010). doi: 10.1016/j.future.2010.02.013StratusLab: Claudia Platform (2013). http://stratuslab.eu/doku.php/claudiaSundareswaran, S., Squicciarini, A., Lin, D.: A brokerage-based approach for cloud service selection. In: Proceedings of the 2012 IEEE 5th International Conference on Cloud Computing, CLOUD ’12, pp. 558–565 (2012). doi: 10.1109/CLOUD.2012.119Telefónica Investigación y Desarrollo S.A. Unipersonal.: Telefónicas TCloud API Specification. (2010). http://www.tid.es/files/doc/apis/TCloud_API_Spec_v0.9.pdfYangui, S., Marshall, I.J., Laisne, J.P., Tata, S.: CompatibleOne: The open source cloud broker. J. Grid Comput. (2013). doi: 10.1007/s10723-013-9285-