HIL: designing an exokernel for the data center

We propose a new Exokernel-like layer to allow mutually untrusting physically deployed services to efficiently share the resources of a data center. We believe that such a layer offers not only efficiency gains, but may also enable new economic models, new applications, and new security-sensitive uses. A prototype (currently in active use) demonstrates that the proposed layer is viable, and can support a variety of existing provisioning tools and use cases.Partial support for this work was provided by the MassTech Collaborative Research Matching Grant Program, National Science Foundation awards 1347525 and 1149232 as well as the several commercial partners of the Massachusetts Open Cloud who may be found at http://www.massopencloud.or

Leveraging Resources on Anonymous Mobile Edge Nodes

Smart devices have become an essential component in the life of mankind. The quick rise of smartphones, IoTs, and wearable devices enabled applications that were not possible few years ago, e.g., health monitoring and online banking. Meanwhile, smart sensing laid the infrastructure for smart homes and smart cities. The intrusive nature of smart devices granted access to huge amounts of raw data. Researchers seized the moment with complex algorithms and data models to process the data over the cloud and extract as much information as possible. However, the pace and amount of data generation, in addition to, networking protocols transmitting data to cloud servers failed short in touching more than 20% of what was generated on the edge of the network. On the other hand, smart devices carry a large set of resources, e.g., CPU, memory, and camera, that sit idle most of the time. Studies showed that for plenty of the time resources are either idle, e.g., sleeping and eating, or underutilized, e.g. inertial sensors during phone calls. These findings articulate a problem in processing large data sets, while having idle resources in the close proximity. In this dissertation, we propose harvesting underutilized edge resources then use them in processing the huge data generated, and currently wasted, through applications running at the edge of the network. We propose flipping the concept of cloud computing, instead of sending massive amounts of data for processing over the cloud, we distribute lightweight applications to process data on users\u27 smart devices. We envision this approach to enhance the network\u27s bandwidth, grant access to larger datasets, provide low latency responses, and more importantly involve up-to-date user\u27s contextual information in processing. However, such benefits come with a set of challenges: How to locate suitable resources? How to match resources with data providers? How to inform resources what to do? and When? How to orchestrate applications\u27 execution on multiple devices? and How to communicate between devices on the edge? Communication between devices at the edge has different parameters in terms of device mobility, topology, and data rate. Standard protocols, e.g., Wi-Fi or Bluetooth, were not designed for edge computing, hence, does not offer a perfect match. Edge computing requires a lightweight protocol that provides quick device discovery, decent data rate, and multicasting to devices in the proximity. Bluetooth features wide acceptance within the IoT community, however, the low data rate and unicast communication limits its use on the edge. Despite being the most suitable communication protocol for edge computing and unlike other protocols, Bluetooth has a closed source code that blocks lower layer in front of all forms of research study, enhancement, and customization. Hence, we offer an open source version of Bluetooth and then customize it for edge computing applications. In this dissertation, we propose Leveraging Resources on Anonymous Mobile Edge Nodes (LAMEN), a three-tier framework where edge devices are clustered by proximities. On having an application to execute, LAMEN clusters discover and allocate resources, share application\u27s executable with resources, and estimate incentives for each participating resource. In a cluster, a single head node, i.e., mediator, is responsible for resource discovery and allocation. Mediators orchestrate cluster resources and present them as a virtually large homogeneous resource. For example, two devices each offering either a camera or a speaker are presented outside the cluster as a single device with both camera and speaker, this can be extended to any combination of resources. Then, mediator handles applications\u27 distribution within a cluster as needed. Also, we provide a communication protocol that is customizable to the edge environment and application\u27s need. Pushing lightweight applications that end devices can execute over their locally generated data have the following benefits: First, avoid sharing user data with cloud server, which is a privacy concern for many of them; Second, introduce mediators as a local cloud controller closer to the edge; Third, hide the user\u27s identity behind mediators; and Finally, enhance bandwidth utilization by keeping raw data at the edge and transmitting processed information. Our evaluation shows an optimized resource lookup and application assignment schemes. In addition to, scalability in handling networks with large number of devices. In order to overcome the communication challenges, we provide an open source communication protocol that we customize for edge computing applications, however, it can be used beyond the scope of LAMEN. Finally, we present three applications to show how LAMEN enables various application domains on the edge of the network. In summary, we propose a framework to orchestrate underutilized resources at the edge of the network towards processing data that are generated in their proximity. Using the approaches explained later in the dissertation, we show how LAMEN enhances the performance of applications and enables a new set of applications that were not feasible

Competitive power control of distributed power plants

Joint Doctoral Programme in Electric Energy Systems : Universidad de Málaga, Universidad de Sevilla, Universidad del País Vasco y Universitat Politècnica de CatalunyaNowadays, the electrical energy sector is currently found in a dramatic changing paradigm, which moves towards an increasing trend in generating power at distribution levels, where electricity is typically consumed, by means of non-conventional/renewable based generation units. These new generation technologies, termed as distributed generation, not only offers a non-pollutant, cheap and efficient source of energy to cover increasing demand, but also enhance the reliability of supply to critical loads and reduce the need for additional grid reinforcements. Aside of the technical benefits provided, distributed generation will massively integrate renewable energy resources, with new type of loads and end-user actors, such as prosumers, demand responsive loads, or electric vehicles. Where these actors will actively participate in energy and auxiliary service markets, depending on their available or constrained energy needs. For this reason, the work presented in this Thesis deals with designing and implementing advanced hierarchical control solutions to renewable-based power plants with the purpose of achieving advanced grid conection performance while reaching maximum economic benefits from its optimum real-time operation. Initially, an extensive analysis on the main renewable-based power plant hierarchical control solutions currently on the shelf, is performed. This study not only covered the specific case of renewable-based power plants, but also advanced microgrid and smart grid control solutions. Once the main renewable-based power plant hierarchical solutions were analized, a novel Hierarchical Distributed Control Structure (HDCS) is proposed for increased management of renewable-based active distributed plants. This hierarchical control structure comprises all possible functional levels from the higher long-term economic scheduling layer, to the instantaneous supervisory control of the resource, emphasizing the entire operation and control functionalities needed for increasing the integration of active distributed power plants. In order to achieve real-time control capabilities in active distribution systems, the present thesis introduces a novel power sharing control strategy, based on the competitive operation of multiple active participating agents (distributed generators, demand response and energy storage systems) through the implementation of market rules. Such control capabilities are satisfied by applying a price control signal over the entire grid control architecture, being the final-end participating agent, the responsible entity in charge of deciding its own generation/demand involvement based on its marginal or affordable electricity costs. In addition, it reduces the information volume to be transmitted and processing requirements, as the higher control levels do not need to have knowledge on the detailed distribution system topology and contributing actors. In order to have a meaningful evaluation of the proposed competitive control capabilities, a wave power plant application has been selected, which constitutes a challenging scenario for the controller itself to achieve advanced real-time control capabilities in such an oscillating renewable energy resource. In order to suitably characterize the wave energy resource profile resulting from maximum energy absorption, this Thesis introduce a novel adaptive vector controller, which maximizes the energy extraction from the resource regardless of the dominant irregular wave frequency characteristics. For the specific wave power plant application considered, the competitive control does not only ensures real-time optimum resource allocation for satisfying a given production objective, but also provides optimum long term operation of the system. As a result, overall plant costs reductions can be achieved under the competitive operation, since the plant scheduled energy is satisfied by making use of the generation units with cheaper cumulative operation costsActualmente, el sector eléctrico se encuentra inmerso en un profundo proceso de restructuración, donde de cada vez más se tiende a generar energía a nivel de distribución, mediante el uso de generación no convencional/renovable. Estas nuevas tecnologías de generación, referidas como generación distribuida, no proporcionan unicamente una fuente de energía no-contaminante, barata y eficiente para cubrir el incremento de demanda, sinó que también pueden proporcionar seguridad de suministro a cargas críticas, así como reducir la necesidad de expansiones futuras de red. Además de las capacidades técnicas proporcionadas, la generación distribuida hará posible la integración masiva de sistemas de generación renovable, con nuevos tipos de cargas y usuarios finales, como prosumidores, cargas regulables, o vehiculos eléctricos, donde todos estos usuarios participaran activamente en mercados de energía y servicios auxiliares, dependiendo de sus requisitos de uso de energía. Por lo tanto, el trabajo realizado en esta tesis se centra en el diseño e implementación de soluciones jerárquicas de control avanzado en plantas de generación renovable, con el objetivo de obtener un comportamiento harmonioso de intercacción con la red, mientras la operación de la planta maximiza los beneficios derivados de su operación en tiempo real. Inicialmente, se ha llevado a cabo una revisión extensa sobre los sistemas de control jerárquico comunmente implementados en plantas de generación renovable, en microredes y en redes inteligentes. Una vez revisados los principales sistemas de control jerárquico en este tipo de aplicaciones, se propone un una novedosa estructura de control, que cubre todos los niveles de control posibles, desde el más alto nivel de gestión económica, hasta el control detallado del recurso de generación. Para lograr capacidades de control en tiempo real en sistemas activos de distribución, la presente tesis propone una nueva estrategia de control de reparto de potencia, basada en la operación competitiva de múltiples agentes participantes activos (generadores distribuidos, respuesta de demanda y sistemas de almacenamiento de energía) mediante la implementación de reglas del mercado. Dichas capacidades de control se satisfacen aplicando una señal de precio a lo largo de toda la arquitectura de control, siendo el agente de final, el ente responsable de decidir su propia participación en la generación/demanda en función de sus propios costes de electricidad marginales o asumibles. Además, reduce el volumen de información a transmitir y los requisitos de procesamiento de datos, ya que los niveles de control más altos no necesitan tener conocimiento sobre la topología del sistema de distribución detallado ni de la contribución de los actores adyacentes. Para llevar a cabo una evaluación significativa de las capacidades del controlador competitivo propuesto, se ha seleccionado una planta de generación undimotriz, como escenario más desfavorable, ya que el controlador debe asegurar un control estable de la potencia inyectada en un escenario altamente oscilante. Con el fin de caracterizar adecuadamente el perfil de recursos de energía de las olas resultante de la máxima absorción de energía, esta Tesis introduce un nuevo controlador de vector adaptativo, que maximiza la extracción de energía del recurso independientemente de las características dominantes de frecuencia de onda irregular. Para la aplicación de la planta de energía de onda específica considerada, el control competitivo no solo garantiza la asignación óptima de recursos en tiempo real para satisfacer un objetivo de producción dado, sino que también proporciona una operación óptima del sistema a largo plazo. Como resultado, se pueden lograr reducciones generales de los costos de la planta en el marco de la operación competitiva, ya que la energía programada de la planta se satisface haciendo uso de las unidadPostprint (published version

Competitive power control of distributed power plants

Nowadays, the electrical energy sector is currently found in a dramatic changing paradigm, which moves towards an increasing trend in generating power at distribution levels, where electricity is typically consumed, by means of non-conventional/renewable based generation units. These new generation technologies, termed as distributed generation, not only offers a non-pollutant, cheap and efficient source of energy to cover increasing demand, but also enhance the reliability of supply to critical loads and reduce the need for additional grid reinforcements. Aside of the technical benefits provided, distributed generation will massively integrate renewable energy resources, with new type of loads and end-user actors, such as prosumers, demand responsive loads, or electric vehicles. Where these actors will actively participate in energy and auxiliary service markets, depending on their available or constrained energy needs. For this reason, the work presented in this Thesis deals with designing and implementing advanced hierarchical control solutions to renewable-based power plants with the purpose of achieving advanced grid conection performance while reaching maximum economic benefits from its optimum real-time operation. Initially, an extensive analysis on the main renewable-based power plant hierarchical control solutions currently on the shelf, is performed. This study not only covered the specific case of renewable-based power plants, but also advanced microgrid and smart grid control solutions. Once the main renewable-based power plant hierarchical solutions were analized, a novel Hierarchical Distributed Control Structure (HDCS) is proposed for increased management of renewable-based active distributed plants. This hierarchical control structure comprises all possible functional levels from the higher long-term economic scheduling layer, to the instantaneous supervisory control of the resource, emphasizing the entire operation and control functionalities needed for increasing the integration of active distributed power plants. In order to achieve real-time control capabilities in active distribution systems, the present thesis introduces a novel power sharing control strategy, based on the competitive operation of multiple active participating agents (distributed generators, demand response and energy storage systems) through the implementation of market rules. Such control capabilities are satisfied by applying a price control signal over the entire grid control architecture, being the final-end participating agent, the responsible entity in charge of deciding its own generation/demand involvement based on its marginal or affordable electricity costs. In addition, it reduces the information volume to be transmitted and processing requirements, as the higher control levels do not need to have knowledge on the detailed distribution system topology and contributing actors. In order to have a meaningful evaluation of the proposed competitive control capabilities, a wave power plant application has been selected, which constitutes a challenging scenario for the controller itself to achieve advanced real-time control capabilities in such an oscillating renewable energy resource. In order to suitably characterize the wave energy resource profile resulting from maximum energy absorption, this Thesis introduce a novel adaptive vector controller, which maximizes the energy extraction from the resource regardless of the dominant irregular wave frequency characteristics. For the specific wave power plant application considered, the competitive control does not only ensures real-time optimum resource allocation for satisfying a given production objective, but also provides optimum long term operation of the system. As a result, overall plant costs reductions can be achieved under the competitive operation, since the plant scheduled energy is satisfied by making use of the generation units with cheaper cumulative operation costsActualmente, el sector eléctrico se encuentra inmerso en un profundo proceso de restructuración, donde de cada vez más se tiende a generar energía a nivel de distribución, mediante el uso de generación no convencional/renovable. Estas nuevas tecnologías de generación, referidas como generación distribuida, no proporcionan unicamente una fuente de energía no-contaminante, barata y eficiente para cubrir el incremento de demanda, sinó que también pueden proporcionar seguridad de suministro a cargas críticas, así como reducir la necesidad de expansiones futuras de red. Además de las capacidades técnicas proporcionadas, la generación distribuida hará posible la integración masiva de sistemas de generación renovable, con nuevos tipos de cargas y usuarios finales, como prosumidores, cargas regulables, o vehiculos eléctricos, donde todos estos usuarios participaran activamente en mercados de energía y servicios auxiliares, dependiendo de sus requisitos de uso de energía. Por lo tanto, el trabajo realizado en esta tesis se centra en el diseño e implementación de soluciones jerárquicas de control avanzado en plantas de generación renovable, con el objetivo de obtener un comportamiento harmonioso de intercacción con la red, mientras la operación de la planta maximiza los beneficios derivados de su operación en tiempo real. Inicialmente, se ha llevado a cabo una revisión extensa sobre los sistemas de control jerárquico comunmente implementados en plantas de generación renovable, en microredes y en redes inteligentes. Una vez revisados los principales sistemas de control jerárquico en este tipo de aplicaciones, se propone un una novedosa estructura de control, que cubre todos los niveles de control posibles, desde el más alto nivel de gestión económica, hasta el control detallado del recurso de generación. Para lograr capacidades de control en tiempo real en sistemas activos de distribución, la presente tesis propone una nueva estrategia de control de reparto de potencia, basada en la operación competitiva de múltiples agentes participantes activos (generadores distribuidos, respuesta de demanda y sistemas de almacenamiento de energía) mediante la implementación de reglas del mercado. Dichas capacidades de control se satisfacen aplicando una señal de precio a lo largo de toda la arquitectura de control, siendo el agente de final, el ente responsable de decidir su propia participación en la generación/demanda en función de sus propios costes de electricidad marginales o asumibles. Además, reduce el volumen de información a transmitir y los requisitos de procesamiento de datos, ya que los niveles de control más altos no necesitan tener conocimiento sobre la topología del sistema de distribución detallado ni de la contribución de los actores adyacentes. Para llevar a cabo una evaluación significativa de las capacidades del controlador competitivo propuesto, se ha seleccionado una planta de generación undimotriz, como escenario más desfavorable, ya que el controlador debe asegurar un control estable de la potencia inyectada en un escenario altamente oscilante. Con el fin de caracterizar adecuadamente el perfil de recursos de energía de las olas resultante de la máxima absorción de energía, esta Tesis introduce un nuevo controlador de vector adaptativo, que maximiza la extracción de energía del recurso independientemente de las características dominantes de frecuencia de onda irregular. Para la aplicación de la planta de energía de onda específica considerada, el control competitivo no solo garantiza la asignación óptima de recursos en tiempo real para satisfacer un objetivo de producción dado, sino que también proporciona una operación óptima del sistema a largo plazo. Como resultado, se pueden lograr reducciones generales de los costos de la planta en el marco de la operación competitiva, ya que la energía programada de la planta se satisface haciendo uso de las unida

The "MIND" Scalable PIM Architecture

MIND (Memory, Intelligence, and Network Device) is an advanced parallel computer architecture for high performance computing and scalable embedded processing. It is a Processor-in-Memory (PIM) architecture integrating both DRAM bit cells and CMOS logic devices on the same silicon die. MIND is multicore with multiple memory/processor nodes on each chip and supports global shared memory across systems of MIND components. MIND is distinguished from other PIM architectures in that it incorporates mechanisms for efficient support of a global parallel execution model based on the semantics of message-driven multithreaded split-transaction processing. MIND is designed to operate either in conjunction with other conventional microprocessors or in standalone arrays of like devices. It also incorporates mechanisms for fault tolerance, real time execution, and active power management. This paper describes the major elements and operational methods of the MIND architecture

High Energy Physics Forum for Computational Excellence: Working Group Reports (I. Applications Software II. Software Libraries and Tools III. Systems)

Computing plays an essential role in all aspects of high energy physics. As computational technology evolves rapidly in new directions, and data throughput and volume continue to follow a steep trend-line, it is important for the HEP community to develop an effective response to a series of expected challenges. In order to help shape the desired response, the HEP Forum for Computational Excellence (HEP-FCE) initiated a roadmap planning activity with two key overlapping drivers -- 1) software effectiveness, and 2) infrastructure and expertise advancement. The HEP-FCE formed three working groups, 1) Applications Software, 2) Software Libraries and Tools, and 3) Systems (including systems software), to provide an overview of the current status of HEP computing and to present findings and opportunities for the desired HEP computational roadmap. The final versions of the reports are combined in this document, and are presented along with introductory material.Comment: 72 page

An Investigation into the Performance Evaluation of Connected Vehicle Applications: From Real-World Experiment to Parallel Simulation Paradigm

A novel system was developed that provides drivers lane merge advisories, using vehicle trajectories obtained through Dedicated Short Range Communication (DSRC). It was successfully tested on a freeway using three vehicles, then targeted for further testing, via simulation. The failure of contemporary simulators to effectively model large, complex urban transportation networks then motivated further research into distributed and parallel traffic simulation. An architecture for a closed-loop, parallel simulator was devised, using a new algorithm that accounts for boundary nodes, traffic signals, intersections, road lengths, traffic density, and counts of lanes; it partitions a sample, Tennessee road network more efficiently than tools like METIS, which increase interprocess communications (IPC) overhead by partitioning more transportation corridors. The simulator uses logarithmic accumulation to synchronize parallel simulations, further reducing IPC. Analyses suggest this eliminates up to one-third of IPC overhead incurred by a linear accumulation model

Energy and Performance: Management of Virtual Machines: Provisioning, Placement, and Consolidation

Cloud computing is a new computing paradigm that offers scalable storage and compute resources to users on demand through Internet. Public cloud providers operate large-scale data centers around the world to handle a large number of users request. However, data centers consume an immense amount of electrical energy that can lead to high operating costs and carbon emissions. One of the most common and effective method in order to reduce energy consumption is Dynamic Virtual Machines Consolidation (DVMC) enabled by the virtualization technology. DVMC dynamically consolidates Virtual Machines (VMs) into the minimum number of active servers and then switches the idle servers into a power-saving mode to save energy. However, maintaining the desired level of Quality-of-Service (QoS) between data centers and their users is critical for satisfying users’ expectations concerning performance. Therefore, the main challenge is to minimize the data center energy consumption while maintaining the required QoS. This thesis address this challenge by presenting novel DVMC approaches to reduce the energy consumption of data centers and improve resource utilization under workload independent quality of service constraints. These approaches can be divided into three main categories: heuristic, meta-heuristic and machine learning. Our first contribution is a heuristic algorithm for solving the DVMC problem. The algorithm uses a linear regression-based prediction model to detect over-loaded servers based on the historical utilization data. Then it migrates some VMs from the over-loaded servers to avoid further performance degradations. Moreover, our algorithm consolidates VMs on fewer number of server for energy saving. The second and third contributions are two novel DVMC algorithms based on the Reinforcement Learning (RL) approach. RL is interesting for highly adaptive and autonomous management in dynamic environments. For this reason, we use RL to solve two main sub-problems in VM consolidation. The first sub-problem is the server power mode detection (sleep or active). The second sub-problem is to find an effective solution for server status detection (overloaded or non-overloaded). The fourth contribution of this thesis is an online optimization meta-heuristic algorithm called Ant Colony System-based Placement Optimization (ACS-PO). ACS is a suitable approach for VM consolidation due to the ease of parallelization, that it is close to the optimal solution, and its polynomial worst-case time complexity. The simulation results show that ACS-PO provides substantial improvement over other heuristic algorithms in reducing energy consumption, the number of VM migrations, and performance degradations. Our fifth contribution is a Hierarchical VM management (HiVM) architecture based on a three-tier data center topology which is very common use in data centers. HiVM has the ability to scale across many thousands of servers with energy efficiency. Our sixth contribution is a Utilization Prediction-aware Best Fit Decreasing (UP-BFD) algorithm. UP-BFD can avoid SLA violations and needless migrations by taking into consideration the current and predicted future resource requirements for allocation, consolidation, and placement of VMs. Finally, the seventh and the last contribution is a novel Self-Adaptive Resource Management System (SARMS) in data centers. To achieve scalability, SARMS uses a hierarchical architecture that is partially inspired from HiVM. Moreover, SARMS provides self-adaptive ability for resource management by dynamically adjusting the utilization thresholds for each server in data centers.Siirretty Doriast