Efficient and elastic management of computing infrastructures

Tesis por compendio[EN] Modern data centers integrate a lot of computer and electronic devices. However, some reports state that the mean usage of a typical data center is around 50% of its peak capacity, and the mean usage of each server is between 10% and 50%. A lot of energy is destined to power on computer hardware that most of the time remains idle. Therefore, it would be possible to save energy simply by powering off those parts from the data center that are not actually used, and powering them on again as they are needed. Most data centers have computing clusters that are used for intensive computing, recently evolving towards an on-premises Cloud service model. Despite the use of low consuming components, higher energy savings can be achieved by dynamically adapting the system to the actual workload. The main approach in this case is the usage of energy saving criteria for scheduling the jobs or the virtual machines into the working nodes. The aim is to power off idle servers automatically. But it is necessary to schedule the power management of the servers in order to minimize the impact on the end users and their applications. The objective of this thesis is the elastic and efficient management of cluster infrastructures, with the aim of reducing the costs associated to idle components. This objective is addressed by automating the power management of the working nodes in a computing cluster, and also proactive stimulating the load distribution to achieve idle resources that could be powered off by means of memory overcommitment and live migration of virtual machines. Moreover, this automation is of interest for virtual clusters, as they also suffer from the same problems. While in physical clusters idle working nodes waste energy, in the case of virtual clusters that are built from virtual machines, the idle working nodes can waste money in commercial Clouds or computational resources in an on-premises Cloud.[ES] En los Centros de Procesos de Datos (CPD) existe una gran concentración de dispositivos informáticos y de equipamiento electrónico. Sin embargo, algunos estudios han mostrado que la utilización media de los CPD está en torno al 50%, y que la utilización media de los servidores se encuentra entre el 10% y el 50%. Estos datos evidencian que existe una gran cantidad de energía destinada a alimentar equipamiento ocioso, y que podríamos conseguir un ahorro energético simplemente apagando los componentes que no se estén utilizando. En muchos CPD suele haber clusters de computadores que se utilizan para computación de altas prestaciones y para la creación de Clouds privados. Si bien se ha tratado de ahorrar energía utilizando componentes de bajo consumo, también es posible conseguirlo adaptando los sistemas a la carga de trabajo en cada momento. En los últimos años han surgido trabajos que investigan la aplicación de criterios energéticos a la hora de seleccionar en qué servidor, de entre los que forman un cluster, se debe ejecutar un trabajo o alojar una máquina virtual. En muchos casos se trata de conseguir equipos ociosos que puedan ser apagados, pero habitualmente se asume que dicho apagado se hace de forma automática, y que los equipos se encienden de nuevo cuando son necesarios. Sin embargo, es necesario hacer una planificación de encendido y apagado de máquinas para minimizar el impacto en el usuario final. En esta tesis nos planteamos la gestión elástica y eficiente de infrastructuras de cálculo tipo cluster, con el objetivo de reducir los costes asociados a los componentes ociosos. Para abordar este problema nos planteamos la automatización del encendido y apagado de máquinas en los clusters, así como la aplicación de técnicas de migración en vivo y de sobreaprovisionamiento de memoria para estimular la obtención de equipos ociosos que puedan ser apagados. Además, esta automatización es de interés para los clusters virtuales, puesto que también sufren el problema de los componentes ociosos, sólo que en este caso están compuestos por, en lugar de equipos físicos que gastan energía, por máquinas virtuales que gastan dinero en un proveedor Cloud comercial o recursos en un Cloud privado.[CA] En els Centres de Processament de Dades (CPD) hi ha una gran concentració de dispositius informàtics i d'equipament electrònic. No obstant això, alguns estudis han mostrat que la utilització mitjana dels CPD està entorn del 50%, i que la utilització mitjana dels servidors es troba entre el 10% i el 50%. Estes dades evidencien que hi ha una gran quantitat d'energia destinada a alimentar equipament ociós, i que podríem aconseguir un estalvi energètic simplement apagant els components que no s'estiguen utilitzant. En molts CPD sol haver-hi clusters de computadors que s'utilitzen per a computació d'altes prestacions i per a la creació de Clouds privats. Si bé s'ha tractat d'estalviar energia utilitzant components de baix consum, també és possible aconseguir-ho adaptant els sistemes a la càrrega de treball en cada moment. En els últims anys han sorgit treballs que investiguen l'aplicació de criteris energètics a l'hora de seleccionar en quin servidor, d'entre els que formen un cluster, s'ha d'executar un treball o allotjar una màquina virtual. En molts casos es tracta d'aconseguir equips ociosos que puguen ser apagats, però habitualment s'assumix que l'apagat es fa de forma automàtica, i que els equips s'encenen novament quan són necessaris. No obstant això, és necessari fer una planificació d'encesa i apagat de màquines per a minimitzar l'impacte en l'usuari final. En esta tesi ens plantegem la gestió elàstica i eficient d'infrastructuras de càlcul tipus cluster, amb l'objectiu de reduir els costos associats als components ociosos. Per a abordar este problema ens plantegem l'automatització de l'encesa i apagat de màquines en els clusters, així com l'aplicació de tècniques de migració en viu i de sobreaprovisionament de memòria per a estimular l'obtenció d'equips ociosos que puguen ser apagats. A més, esta automatització és d'interés per als clusters virtuals, ja que també patixen el problema dels components ociosos, encara que en este cas estan compostos per, en compte d'equips físics que gasten energia, per màquines virtuals que gasten diners en un proveïdor Cloud comercial o recursos en un Cloud privat.Alfonso Laguna, CD. (2015). Efficient and elastic management of computing infrastructures [Tesis doctoral no publicada]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/57187Compendi

Digitising the Industry Internet of Things Connecting the Physical, Digital and VirtualWorlds

This book provides an overview of the current Internet of Things (IoT) landscape, ranging from the research, innovation and development priorities to enabling technologies in a global context. A successful deployment of IoT technologies requires integration on all layers, be it cognitive and semantic aspects, middleware components, services, edge devices/machines and infrastructures. It is intended to be a standalone book in a series that covers the Internet of Things activities of the IERC - Internet of Things European Research Cluster from research to technological innovation, validation and deployment. The book builds on the ideas put forward by the European Research Cluster and the IoT European Platform Initiative (IoT-EPI) and presents global views and state of the art results on the challenges facing the research, innovation, development and deployment of IoT in the next years. The IoT is bridging the physical world with virtual world and requires sound information processing capabilities for the "digital shadows" of these real things. The research and innovation in nanoelectronics, semiconductor, sensors/actuators, communication, analytics technologies, cyber-physical systems, software, swarm intelligent and deep learning systems are essential for the successful deployment of IoT applications. The emergence of IoT platforms with multiple functionalities enables rapid development and lower costs by offering standardised components that can be shared across multiple solutions in many industry verticals. The IoT applications will gradually move from vertical, single purpose solutions to multi-purpose and collaborative applications interacting across industry verticals, organisations and people, being one of the essential paradigms of the digital economy. Many of those applications still have to be identified and involvement of end-users including the creative sector in this innovation is crucial. The IoT applications and deployments as integrated building blocks of the new digital economy are part of the accompanying IoT policy framework to address issues of horizontal nature and common interest (i.e. privacy, end-to-end security, user acceptance, societal, ethical aspects and legal issues) for providing trusted IoT solutions in a coordinated and consolidated manner across the IoT activities and pilots. In this, context IoT ecosystems offer solutions beyond a platform and solve important technical challenges in the different verticals and across verticals. These IoT technology ecosystems are instrumental for the deployment of large pilots and can easily be connected to or build upon the core IoT solutions for different applications in order to expand the system of use and allow new and even unanticipated IoT end uses. Technical topics discussed in the book include: • Introduction• Digitising industry and IoT as key enabler in the new era of Digital Economy• IoT Strategic Research and Innovation Agenda• IoT in the digital industrial context: Digital Single Market• Integration of heterogeneous systems and bridging the virtual, digital and physical worlds• Federated IoT platforms and interoperability• Evolution from intelligent devices to connected systems of systems by adding new layers of cognitive behaviour, artificial intelligence and user interfaces.• Innovation through IoT ecosystems• Trust-based IoT end-to-end security, privacy framework• User acceptance, societal, ethical aspects and legal issues• Internet of Things Application

Digitising the Industry Internet of Things Connecting the Physical, Digital and VirtualWorlds

This book provides an overview of the current Internet of Things (IoT) landscape, ranging from the research, innovation and development priorities to enabling technologies in a global context. A successful deployment of IoT technologies requires integration on all layers, be it cognitive and semantic aspects, middleware components, services, edge devices/machines and infrastructures. It is intended to be a standalone book in a series that covers the Internet of Things activities of the IERC - Internet of Things European Research Cluster from research to technological innovation, validation and deployment. The book builds on the ideas put forward by the European Research Cluster and the IoT European Platform Initiative (IoT-EPI) and presents global views and state of the art results on the challenges facing the research, innovation, development and deployment of IoT in the next years. The IoT is bridging the physical world with virtual world and requires sound information processing capabilities for the "digital shadows" of these real things. The research and innovation in nanoelectronics, semiconductor, sensors/actuators, communication, analytics technologies, cyber-physical systems, software, swarm intelligent and deep learning systems are essential for the successful deployment of IoT applications. The emergence of IoT platforms with multiple functionalities enables rapid development and lower costs by offering standardised components that can be shared across multiple solutions in many industry verticals. The IoT applications will gradually move from vertical, single purpose solutions to multi-purpose and collaborative applications interacting across industry verticals, organisations and people, being one of the essential paradigms of the digital economy. Many of those applications still have to be identified and involvement of end-users including the creative sector in this innovation is crucial. The IoT applications and deployments as integrated building blocks of the new digital economy are part of the accompanying IoT policy framework to address issues of horizontal nature and common interest (i.e. privacy, end-to-end security, user acceptance, societal, ethical aspects and legal issues) for providing trusted IoT solutions in a coordinated and consolidated manner across the IoT activities and pilots. In this, context IoT ecosystems offer solutions beyond a platform and solve important technical challenges in the different verticals and across verticals. These IoT technology ecosystems are instrumental for the deployment of large pilots and can easily be connected to or build upon the core IoT solutions for different applications in order to expand the system of use and allow new and even unanticipated IoT end uses. Technical topics discussed in the book include: • Introduction• Digitising industry and IoT as key enabler in the new era of Digital Economy• IoT Strategic Research and Innovation Agenda• IoT in the digital industrial context: Digital Single Market• Integration of heterogeneous systems and bridging the virtual, digital and physical worlds• Federated IoT platforms and interoperability• Evolution from intelligent devices to connected systems of systems by adding new layers of cognitive behaviour, artificial intelligence and user interfaces.• Innovation through IoT ecosystems• Trust-based IoT end-to-end security, privacy framework• User acceptance, societal, ethical aspects and legal issues• Internet of Things Application

A proposed model to analyse risk and return for a large computing system adoption

This thesis presents Organisational Sustainability Modelling (OSM), a new method to model and analyse risk and return systematically for the adoption of large systems such as Cloud Computing. Return includes improvements in technical efficiency, profitability and service. Risk includes controlled risk (risk-control rate) and uncontrolled risk (beta), although uncontrolled risk cannot be evaluated directly. Three OSM metrics, actual return value, expected return value and risk-control rate are used to calculate uncontrolled risk. The OSM data collection process in which hundreds of datasets (rows of data containing three OSM metrics in each row) are used as inputs is explained. Outputs including standard error, mean squared error, Durbin-Watson, p-value and R-squared value are calculated. Visualisation is used to illustrate quality and accuracy of data analysis. The metrics, process and interpretation of data analysis is presented and the rationale is explained in the review of the OSM method.Three case studies are used to illustrate the validity of OSM:• National Health Service (NHS) is a technical application concerned with backing up data files and focuses on improvement in efficiency.• Vodafone/Apple is a cost application and focuses on profitability.• The iSolutions Group, University of Southampton focuses on service improvement using user feedback.The NHS case study is explained in detail. The expected execution time calculated by OSM to complete all backup activity in Cloud-based systems matches actual execution time to within 0.01%. The Cloud system shows improved efficiency in both sets of comparisons. All three case studies confirm there are benefits for the adoption of a large computer system such as the Cloud. Together these demonstrations answer the two research questions for this thesis:1. How do you model and analyse risk and return on adoption of large computing systems systematically and coherently?2. Can the same method be used in risk mitigation of system adoption?Limitations of this study, a reproducibility case, comparisons with similar approaches, research contributions and future work are also presented

Energy Efficiency

Energy efficiency is finally a common sense term. Nowadays almost everyone knows that using energy more efficiently saves money, reduces the emissions of greenhouse gasses and lowers dependence on imported fossil fuels. We are living in a fossil age at the peak of its strength. Competition for securing resources for fuelling economic development is increasing, price of fuels will increase while availability of would gradually decline. Small nations will be first to suffer if caught unprepared in the midst of the struggle for resources among the large players. Here it is where energy efficiency has a potential to lead toward the natural next step - transition away from imported fossil fuels! Someone said that the only thing more harmful then fossil fuel is fossilized thinking. It is our sincere hope that some of chapters in this book will influence you to take a fresh look at the transition to low carbon economy and the role that energy efficiency can play in that process

Proceedings of the Conference on Production Systems and Logistics: CPSL 2022

[no abstract available

Using MapReduce Streaming for Distributed Life Simulation on the Cloud

Distributed software simulations are indispensable in the study of large-scale life models but often require the use of technically complex lower-level distributed computing frameworks, such as MPI. We propose to overcome the complexity challenge by applying the emerging MapReduce (MR) model to distributed life simulations and by running such simulations on the cloud. Technically, we design optimized MR streaming algorithms for discrete and continuous versions of Conway’s life according to a general MR streaming pattern. We chose life because it is simple enough as a testbed for MR’s applicability to a-life simulations and general enough to make our results applicable to various lattice-based a-life models. We implement and empirically evaluate our algorithms’ performance on Amazon’s Elastic MR cloud. Our experiments demonstrate that a single MR optimization technique called strip partitioning can reduce the execution time of continuous life simulations by 64%. To the best of our knowledge, we are the first to propose and evaluate MR streaming algorithms for lattice-based simulations. Our algorithms can serve as prototypes in the development of novel MR simulation algorithms for large-scale lattice-based a-life models.https://digitalcommons.chapman.edu/scs_books/1014/thumbnail.jp