Runtime virtual machine recontextualization for clouds

We introduce and define the concept of recontextualization for cloud applications by extending contextualization, i.e. the dynamic configuration of virtual machines (VM) upon initialization, with autonomous updates during runtime. Recontextualization allows VM images and instances to be dynamically re-configured without restarts or downtime, and the concept is applicable to all aspects of configuring a VM from virtual hardware to multi-tier software stacks. Moreover, we propose a runtime cloud recontextualization mechanism based on virtual device management that enables recontextualization without the need to customize the guest VM. We illustrate our concept and validate our mechanism via a use case demonstration: the reconfiguration of a cross-cloud migratable monitoring service in a dynamic cloud environment. We discuss the details of the interoperable recontextualization mechanism, its architecture and demonstrate a proof of concept implementation. A performance evaluation illustrates the feasibility of the approach and shows that the recontextualization mechanism performs adequately with an overhead of 18% of the total migration time

A platform to deploy customized scientific virtual infrastructures on the cloud

This paper presents a software platform to dynamically deploy complex scientific virtual computing infrastructures, on top of Infrastructure as a Service (IaaS) Clouds. The platform orchestrates different services to provision the virtual computing resources. It dynamically installs the appropriate software to satisfy the requirements of a researcher, both on public and on-premise Clouds. The platform provides a web interface to enable the users to easily management of the lifecycle of virtual infrastructures. It enables users to define infrastructures, share them with other users, deploy and relinquish them, add or remove resources dynamically, create and share application recipes, etc. The paper also describes three case studies to deploy complex infrastructures, namely a Hadoop cluster, a single-node to perform NGS sequencing and a gateway for users to access the European Grid Infrastructure (EGI). This platform promotes a better use of on-premise hardware resources of a research center by allocating the computing resources just-in-time to the specific life time of the virtual infrastructures as well as the deployment of the very same infrastructures on a public Cloud.The authors would to thank the Spanish "Ministerio de Economia y Competitividad" for the project "Clusters Virtuales Elasticos y Migrables sobre Infraestructuras Cloud Hibridas" with reference TIN2013-44390-R.Caballer Fernández, M.; Segrelles Quilis, JD.; Moltó, G.; Blanquer Espert, I. (2015). A platform to deploy customized scientific virtual infrastructures on the cloud. Concurrency and Computation: Practice and Experience. 27(16):4318-4329. https://doi.org/10.1002/cpe.3518S431843292716Mell P Grance T The NIST definition of Cloud computing. NIST Special Publication 800-145 (Final) Technical Report 2011 http://csrc.nist.gov/publications/nistpubs/800-145/SP800-145.pdfBuyya, R., Broberg, J., & Goscinski, A. (Eds.). (2011). Cloud Computing. doi:10.1002/9780470940105Sahoo J Mohapatra S Lath R Virtualization: a survey on concepts, taxonomy and associated security issues 2010 Second International Conference on Computer and Network Technology Bangkok, Thailand 2010 222 226OpenStack OpenStack 2013 http://openstack.orgNurmi D Wolski R Grzegorczyk C Obertelli G Soman S Youseff L Zagorodnov D The Eucalyptus open-source Cloud-computing system Proceedings of 9th IEEE International Symposium on Cluster Computing and the Grid Shanghai, China 2009 124 131Amazon Web Services AWS CloudFormation http://aws.amazon.com/cloudformation/Amazon Web Services AWS OpsWorks http://aws.amazon.com/opsworks/Keahey K Freeman T Contextualization: providing one-click virtual clusters Fourth IEEE International Conference on eScience Indianapolis, Indiana, USA 2008 301 308Keahey K Freeman T Architecting a large-scale elastic environment: recontextualization and adaptive Cloud services for scientific computing 2012Marshall P Keahey K Freeman T Elastic site: using Clouds to elastically extend site resources Proceedings of the 2010 IEEE/ACM 10th International Conference on Cluster, Cloud and Grid Computing CCGRID '10 IEEE Computer Society, Washington, DC, USA 2010 43 52Bresnahan J Freeman T LaBissoniere D Keahey K Managing appliance launches in infrastructure Clouds Proceedings of the 2011 TeraGrid Conference: Extreme Digital Discovery TG '11 ACM, New York, NY, USA 2011 12:1 12:7Apache Whirr 2013 from:http://whirr.apache.org/Juve G Deelman E Automating application deployment in infrastructure clouds Proceedings of the 2011 IEEE Third International Conference on Cloud Computing Technology and Science CLOUDCOM '11 IEEE Computer Society, Washington, DC, USA 2011 658 665OASIS Topology and orchestration specification for cloud applications version 1.0 2013 http://docs.oasis-open.org/tosca/TOSCA/v1.0/TOSCA-v1.0.htmlBinz T Breitenbcher U Haupt F Kopp O Leymann F Nowak A Wagner S OpenTOSCA - a runtime for TOSCA-based cloud applications ICSOC, Lecture Notes in Computer Science 8274 Springer 2013 692 695Puppet Labs IT automation software for system administrators 2013 http://www.puppetlabs.com/Opscode Chef 2013 http://www.opscode.com/chef/DeHaan M Ansible 2013 http://ansible.cc/Vogels, W. (2008). Beyond server consolidation. Queue, 6(1), 20. doi:10.1145/1348583.1348590Carrión JV Moltó G De Alfonso C Caballer M Hernández V A generic catalog and repository service for virtual machine images 2nd International ICST Conference on Cloud Computing (CloudComp 2010) Barcelona, Spain 2010 1 15de Alfonso C Caballer M Alvarruiz F Molto G Hernández V Infrastructure deployment over the Cloud 2011 IEEE Third International Conference on Cloud Computing Technology and Science Athens, Greece 2011 517 521Caballer, M., Blanquer, I., Moltó, G., & de Alfonso, C. (2014). Dynamic Management of Virtual Infrastructures. Journal of Grid Computing, 13(1), 53-70. doi:10.1007/s10723-014-9296-5Dean, J., & Ghemawat, S. (2008). MapReduce. Communications of the ACM, 51(1), 107. doi:10.1145/1327452.1327492Shvachko K Kuang H Radia S Chansler R The Hadoop distributed file system 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST) Incline Village, NV, USA 2010 1 10Altschul, S. F., Gish, W., Miller, W., Myers, E. W., & Lipman, D. J. (1990). Basic local alignment search tool. Journal of Molecular Biology, 215(3), 403-410. doi:10.1016/s0022-2836(05)80360-

Contextualization: dynamic configuration of virtual machines

New VM instances are created from static templates that contain the basic configuration of the VM to achieve elasticity with regards to capacity. Instance specific settings can be injected into the VM during the deployment phase through means of contextualization. So far this is limited to a single data source and data remains static throughout the lifecycle of the VM. We present a layered approach to contextualization that supports different classes of contextualization data available from several sources. The settings are made available to the VM through virtual devices. Inside each VM data from different classes are layered on top of each other to create a unified file hierarchy. Context data can be modified during runtime by updating the contents of the virtual devices, making our approach the first contextualization approach to natively support recontextualization. Recontextualization enables runtime reconfiguration of an executing service and can act as a trigger and key enabler of self-* techniques. This trigger provides a service with a mechanism to adapt or optimize itself in response to a changing environment. The runtime reconfiguration using recontextualization and its potential gains are illustrated in an example with a distributed file system, demonstrating the feasibility of our approach

Energy efficiency embedded service lifecycle: Towards an energy efficient cloud computing architecture

The paper argues the need to provide novel methods and tools to support software developers aiming to optimise energy efficiency and minimise the carbon footprint resulting from designing, developing, deploying and running software in Clouds, while maintaining other quality aspects of software to adequate and agreed levels. A cloud architecture to support energy efficiency at service construction, deployment, and operation is discussed, as well as its implementation and evaluation plans.Postprint (published version

Towards energy aware cloud computing application construction

The energy consumption of cloud computing continues to be an area of significant concern as data center growth continues to increase. This paper reports on an energy efficient interoperable cloud architecture realised as a cloud toolbox that focuses on reducing the energy consumption of cloud applications holistically across all deployment models. The architecture supports energy efficiency at service construction, deployment and operation. We discuss our practical experience during implementation of an architectural component, the Virtual Machine Image Constructor (VMIC), required to facilitate construction of energy aware cloud applications. We carry out a performance evaluation of the component on a cloud testbed. The results show the performance of Virtual Machine construction, primarily limited by available I/O, to be adequate for agile, energy aware software development. We conclude that the implementation of the VMIC is feasible, incurs minimal performance overhead comparatively to the time taken by other aspects of the cloud application construction life-cycle, and make recommendations on enhancing its performance

Dynamic management of virtual infrastructures

The final publication is available at Springer via http://dx.doi.org/10.1007/s10723-014-9296-5Cloud infrastructures are becoming an appropriate solution to address the computational needs of scientific applications. However, the use of public or on-premises Infrastructure as a Service (IaaS) clouds requires users to have non-trivial system administration skills. Resource provisioning systems provide facilities to choose the most suitable Virtual Machine Images (VMI) and basic configuration of multiple instances and subnetworks. Other tasks such as the configuration of cluster services, computational frameworks or specific applications are not trivial on the cloud, and normally users have to manually select the VMI that best fits, including undesired additional services and software packages. This paper presents a set of components that ease the access and the usability of IaaS clouds by automating the VMI selection, deployment, configuration, software installation, monitoring and update of Virtual Appliances. It supports APIs from a large number of virtual platforms, making user applications cloud-agnostic. In addition it integrates a contextualization system to enable the installation and configuration of all the user required applications providing the user with a fully functional infrastructure. Therefore, golden VMIs and configuration recipes can be easily reused across different deployments. Moreover, the contextualization agent included in the framework supports horizontal (increase/decrease the number of resources) and vertical (increase/decrease resources within a running Virtual Machine) by properly reconfiguring the software installed, considering the configuration of the multiple resources running. This paves the way for automatic virtual infrastructure deployment, customization and elastic modification at runtime for IaaS clouds.The authors would like to thank to thank the financial support received from the Ministerio de Economia y Competitividad for the project CodeCloud (TIN2010-17804).Caballer Fernández, M.; Blanquer Espert, I.; Moltó, G.; Alfonso Laguna, CD. (2015). Dynamic management of virtual infrastructures. Journal of Grid Computing. 13(1):53-70. https://doi.org/10.1007/s10723-014-9296-5S5370131de Alfonso, C., Caballer, M., Alvarruiz, F., Molto, G., Hernández, V.: Infrastructure deployment over the cloud. In: 2011 IEEE 3rd International Conference on Cloud Computing Technology and Science, pp. 517–521. IEEE. (2011). doi: 10.1109/CloudCom.2011.77Alvarruiz, F., De Alfonso, C., Caballer, M., Hernández, V.: An energy manager for high performance computer clusters. In: 2012 IEEE 10th International Symposium on Parallel and Distributed Processing with Applications, pp. 231–238. (2012). doi: 10.1109/ISPA.2012.38Amazon Web Services: AWS CloudFormation. (2013). http://aws.amazon.com/es/cloudformation/Apache: Whirr (2013). http://whirr.apache.org/Blanquer, I., Brasche, G., Lezzi, D.: Requirements of scientific applications in cloud offerings. In: Proceedings of the 2012 6th Iberian Grid Infrastructure Conference, IBERGRID ’12, pp. 173–182 (2012)Bresnahan, J., Freeman, T., LaBissoniere, D., Keahey, K.: Managing appliance launches in infrastructure clouds. In: Proceedings of the 2011 TeraGrid Conference: Extreme Digital Discovery, TG ’11, pp. 12:1–12:7. ACM, New York (2011). doi: 10.1145/2016741.2016755Buyya, R., Ranjan, R., Calheiros, R.N.: InterCloud: utility-oriented federation of cloud computing environments for scaling of application services. Algoritm. Archit. Parallel Process. 6081, 20 (2010)Buyya, R., Yeo, C.S., Venugopal, S., Broberg, J., Brandic, I.: Cloud computing and emerging IT platforms: Vision, hype, and reality for delivering computing as the 5th utility. Futur. Gener. Comput. Syst. 25(6), 599–616 (2009). doi: 10.1016/j.future.2008.12.001Caballer, M., De Alfonso, C., Alvarruiz, F., Moltó, G.: EC3: elastic cloud computing cluster. J. Comput. Syst. Sci. (2013). doi: 10.1016/j.jcss.2013.06.005Caballer, M., García, A., Moltó, G., de Alfonso, C.: Towards SLA-driven management of cloud infrastructures to elastically execute scientific applications. In: 6th Iberian Grid Infrastructure Conference (IberGrid), pp. 207–218 (2012)Carrión, J.V., Moltó, G., De Alfonso, C., Caballer, M., Hernández, V.: A generic catalog and repository service for virtual machine images. In: 2nd International ICST Conference on Cloud Computing CloudComp 2010 (2010)Cuomo, A., Modica, G., Distefano, S., Puliafito, A., Rak, M., Tomarchio, O., Venticinque, S., Villano, U.: An SLA-based broker for cloud infrastructures. J. Grid Comput 11(1), 1–25 (2012). doi: 10.1007/s10723-012-9241-4DeHaan, M.: Ansible. http://ansible.cc/ (2013)Distributed Management Task Force, Inc: Open Virtualization Format (OVF) (2010). http://dmtf.org/sites/default/files/standards/documents/DSP0243_1.1.0.pdfDistributed Management Task Force, Inc: Cloud Infrastructure Management Interface (CIMI) Model and REST Interface over HTTP Specification (2012). http://dmtf.org/sites/default/files/standards/documents/DSP0263_1.0.1.pdfEGI.eu: Seeking new horizons: EGI’s role for 2020. Tech. rep. (2012). https://documents.egi.eu/public/RetrieveFile?docid=1098&version=4&filename=EGI-1098-D230-final.pdfElmroth, E., Tordsson, J., Hernández, F.: Self-management challenges for multi-cloud architectures. Towards a service-based internet. Lect. Notes Comput. Sci. 6994, 38–49 (2011)HashiCorp: Vagrant (2013). http://www.vagrantup.com/Jacob, A.: Infrastructure in the cloud era. In: Proceedings of the 2009 International OReilly Conference Velocity (2009)Juve, G., Deelman, E.: Automating application deployment in infrastructure clouds. In: Proceedings of the 2011 IEEE 3rd International Conference on Cloud Computing Technology and Science, CLOUDCOM ’11, pp. 658–665. IEEE Computer Society, Washington DC (2011). doi: 10.1109/CloudCom.2011.102Keahey, K., Freeman, T.: Contextualization: providing one-click virtual clusters. In: 4th IEEE International Conference on eScience, pp. 301–308 (2008)Keahey, K., Freeman, T.: Architecting a large-scale elastic environment: recontextualization and adaptive cloud services for scientific computing (2012)Kecskemeti, G., Kertesz, A., Marosi, A., Kacsuk, P.: Interoperable resource management for establishing federated clouds. In: Achieving Federated and SelfManageable Cloud Infrastructures Theory and Practice, pp. 18–35 (2012). doi: 10.4018/978-1-4666-1631-8.ch002Kertesz, A., Kecskemeti, G., Oriol, M., Kotcauer, P., Acs, S., Rodríguez, M., Mercè, O., Marosi, A.C., Marco, J., Franch, X.: Enhancing federated cloud management with an integrated service monitoring approach. J. Grid Comput. 11(4), 699–720 (2013). doi: 10.1007/s10723-013-9269-0Loutas, N., Kamateri, E., Bosi, F., Tarabanis, K.: Cloud computing interoperability: the state of play. 2011 IEEE 3rd International Conference on Cloud Computing Technology and Science, pp. 752–757 (2011). doi: 10.1109/CloudCom.2011.116Marshall, P., Keahey, K., Freeman, T.: Elastic site: using clouds to elastically extend site resources. In: Proceedings of the 2010 IEEE/ACM 10th International Conference on Cluster, Cloud and Grid Computing, CCGRID ’10, pp. 43–52. IEEE Computer Society, Washington DC (2010). doi: 10.1109/CCGRID.2010.80Massie, M.L., Chun, B.N., Culler, D.E.: The ganglia distributed monitoring system: design, implementation, and experience. Parallel Comput. 30(5-6), 817–840 (2004)Mell, P., Grance, T.: The NIST definition of cloud computing. NIST Special Publication 800-145 (Final). Tech. rep. (2011). http://csrc.nist.gov/publications/nistpubs/800-145/SP800-145.pdfMoltó, G., Caballer, M., Romero, E., Alfonso, C.D.: Elastic memory management of virtualized infrastructures for applications with dynamic memory requirements. In: Proceedings of the International Conference on Computational Science ICCS 2013, pp. 159–168. Elsevier (2013). doi: 10.1016/j.procs.2013.05.179Morfeo: Claudia (2013). http://claudia.morfeo-project.org/wiki/index.php/Main_PageOASIS: Topology and Orchestration Specification for Cloud Applications Version 1.0 (2013). http://docs.oasis-open.org/tosca/TOSCA/v1.0/TOSCA-v1.0.htmlOCCI working group within the Open Grid Forum: Open Cloud Computing Interface Infrastructure (2011). http://ogf.org/documents/GFD.184.pdfOpscode: Chef (2013). http://www.opscode.com/chef/Pawluk, P., Simmons, B., Smit, M., Litoiu, M., Mankovski, S.: Introducing STRATOS: a cloud broker service. In: 2012 IEEE 5th International Conference on Cloud Computing, pp. 891–898 (2012). doi: 10.1109/CLOUD.2012.24Puppet Labs: IT Automation Software for System Administrators (2013). http://www.puppetlabs.com/Redl, C., Breskovic, I., Brandic, I., Dustdar, S.: Automatic SLA matching and provider selection in grid and cloud computing markets. In: Proceedings of the 2012 ACM/IEEE 13th International Conference on Grid Computing, GRID ’12, pp. 85–94. IEEE Computer Society, Washington (2012). doi: 10.1109/Grid.2012.18Rodero-Merino, L., Vaquero, L.M., Gil, V., Galán, F., Fontán, J., Montero, R.S., Llorente, I.M.: From infrastructure delivery to service management in clouds. Futur. Gener. Comput. Syst. 26(8), 1226–1240 (2010). doi: 10.1016/j.future.2010.02.013StratusLab: Claudia Platform (2013). http://stratuslab.eu/doku.php/claudiaSundareswaran, S., Squicciarini, A., Lin, D.: A brokerage-based approach for cloud service selection. In: Proceedings of the 2012 IEEE 5th International Conference on Cloud Computing, CLOUD ’12, pp. 558–565 (2012). doi: 10.1109/CLOUD.2012.119Telefónica Investigación y Desarrollo S.A. Unipersonal.: Telefónicas TCloud API Specification. (2010). http://www.tid.es/files/doc/apis/TCloud_API_Spec_v0.9.pdfYangui, S., Marshall, I.J., Laisne, J.P., Tata, S.: CompatibleOne: The open source cloud broker. J. Grid Comput. (2013). doi: 10.1007/s10723-013-9285-

CodeCloud: A platform to enable execution of programming models on the Clouds

This paper presents a platform that supports the execution of scientific applications covering different programming models (such as Master/Slave, Parallel/MPI, MapReduce and Workflows) on Cloud infrastructures. The platform includes (i) a high-level declarative language to express the requirements of the applications featuring software customization at runtime, (ii) an approach based on virtual containers to encapsulate the logic of the different programming models, (iii) an infrastructure manager to interact with different IaaS backends, (iv) a configuration software to dynamically configure the provisioned resources and (v) a catalog and repository of virtual machine images. By using this platform, an application developer can adapt, deploy and execute parallel applications agnostic to the Cloud backend.The authors wish to thank the financial support received from both the Spanish Ministry of Economy and Competitiveness to develop the project "Servicios avanzados para el despliegue y contextualizacion de aplicaciones virtualizadas para dar soporte a modelos de programacion en entornos cloud", with reference TIN2010-17804.Caballer Fernández, M.; Alfonso Laguna, CD.; Moltó, G.; Romero Alcalde, E.; Blanquer Espert, I.; García García, A. (2014). CodeCloud: A platform to enable execution of programming models on the Clouds. Journal of Systems and Software. 93:187-198. https://doi.org/10.1016/j.jss.2014.02.005S1871989

Energy efficiency embedded service lifecycle: Towards an energy efficient cloud computing architecture

The paper argues the need to provide novel methods and tools to support software developers aiming to optimise energy efficiency and minimise the carbon footprint resulting from designing, developing, deploying and running software in Clouds, while maintaining other quality aspects of software to adequate and agreed levels. A cloud architecture to support energy efficiency at service construction, deployment, and operation is discussed, as well as its implementation and evaluation plans

Prebaked µVMs: Scalable, Instant VM Startup for IaaS Clouds

Abstract-IaaS clouds promise instantaneously available resources to elastic applications. In practice, however, virtual machine (VM) startup times are in the order of several minutes, or at best, several tens of seconds, negatively impacting the elasticity of applications like Web servers that need to scale out to handle dynamically increasing load. VM startup time is strongly influenced by booting the VM's operating system. In this work, we propose using so-called prebaked µVMs to speed up VM startup. µVMs are snapshots of minimal VMs that can be quickly resumed and then configured to application needs by hot-plugging resources. To serve µVMs, we extend our VM boot cache service, Squirrel, allowing to store µVMs for large numbers of VM images on the hosts of a data center. Our experiments show that µVMs can start up in less than one second on a standard file system. Using 1000+ VM images from a production cloud, we show that the respective µVMs can be stored in a compressed and deduplicated file system within 50 GB storage per host, while starting up within 2-3 seconds on average

Towards an interoperable energy efficient Cloud computing architecture-practice & experience

The energy consumption of Cloud computing continues to be an area of significant concern as data center growth continues to increase. This paper reports on an energy efficient interoperable Cloud architecture realized as a Cloud toolbox that focuses on reducing the energy consumption of Cloud applications holistically across all deployments models. The architecture supports energy efficiency at service construction, deployment, and operation and interoperability through the use of the Open Virtualization Format (OVF) standard. We discuss our practical experience during implementation and present an initial performance evaluation of the architecture. The results show that the implementing Cloud provider interoperability is feasible and incurs minimal performance overhead during application deployment in comparison to the time taken to instantiate Virtual Machines