Self-management for large-scale distributed systems

Autonomic computing aims at making computing systems self-managing by using autonomic managers in order to reduce obstacles caused by management complexity. This thesis presents results of research on self-management for large-scale distributed systems. This research was motivated by the increasing complexity of computing systems and their management. In the first part, we present our platform, called Niche, for programming self-managing component-based distributed applications. In our work on Niche, we have faced and addressed the following four challenges in achieving self-management in a dynamic environment characterized by volatile resources and high churn: resource discovery, robust and efficient sensing and actuation, management bottleneck, and scale. We present results of our research on addressing the above challenges. Niche implements the autonomic computing architecture, proposed by IBM, in a fully decentralized way. Niche supports a network-transparent view of the system architecture simplifying the design of distributed self-management. Niche provides a concise and expressive API for self-management. The implementation of the platform relies on the scalability and robustness of structured overlay networks. We proceed by presenting a methodology for designing the management part of a distributed self-managing application. We define design steps that include partitioning of management functions and orchestration of multiple autonomic managers. In the second part, we discuss robustness of management and data consistency, which are necessary in a distributed system. Dealing with the effect of churn on management increases the complexity of the management logic and thus makes its development time consuming and error prone. We propose the abstraction of Robust Management Elements, which are able to heal themselves under continuous churn. Our approach is based on replicating a management element using finite state machine replication with a reconfigurable replica set. Our algorithm automates the reconfiguration (migration) of the replica set in order to tolerate continuous churn. For data consistency, we propose a majority-based distributed key-value store supporting multiple consistency levels that is based on a peer-to-peer network. The store enables the tradeoff between high availability and data consistency. Using majority allows avoiding potential drawbacks of a master-based consistency control, namely, a single-point of failure and a potential performance bottleneck. In the third part, we investigate self-management for Cloud-based storage systems with the focus on elasticity control using elements of control theory and machine learning. We have conducted research on a number of different designs of an elasticity controller, including a State-Space feedback controller and a controller that combines feedback and feedforward control. We describe our experience in designing an elasticity controller for a Cloud-based key-value store using state-space model that enables to trade-off performance for cost. We describe the steps in designing an elasticity controller. We continue by presenting the design and evaluation of ElastMan, an elasticity controller for Cloud-based elastic key-value stores that combines feedforward and feedback control

Distributed aop middleware for large-scale scenarios

En aquesta tesi doctoral presentem una proposta de middleware distribuït pel desenvolupament d'aplicacions de gran escala. La nostra motivació principal és permetre que les responsabilitats distribuïdes d'aquestes aplicacions, com per exemple la replicació, puguin integrar-se de forma transparent i independent. El nostre enfoc es basa en la implementació d'aquestes responsabilitats mitjançant el paradigma d'aspectes distribuïts i es beneficia dels substrats de les xarxes peer-to-peer (P2P) i de la programació orientada a aspectes (AOP) per realitzar-ho de forma descentralitzada, desacoblada, eficient i transparent. La nostra arquitectura middleware es divideix en dues capes: un model de composició i una plataforma escalable de desplegament d'aspectes distribuïts. Per últim, es demostra la viabilitat i aplicabilitat del nostre model mitjançant la implementació i experimentació de prototipus en xarxes de gran escala reals.In this PhD dissertation we present a distributed middleware proposal for large-scale application development. Our main aim is to separate the distributed concerns of these applications, like replication, which can be integrated independently and transparently. Our approach is based on the implementation of these concerns using the paradigm of distributed aspects. In addition, our proposal benefits from the peer-to-peer (P2P) networks and aspect-oriented programming (AOP) substrates to provide these concerns in a decentralized, decoupled, efficient, and transparent way. Our middleware architecture is divided into two layers: a composition model and a scalable deployment platform for distributed aspects. Finally, we demonstrate the viability and applicability of our model via implementation and experimentation of prototypes in real large-scale networks

Data Centric Peer-to-Peer Communication in Power Grids

We study the use of peer-to-peer based declarative data management to enable efficient monitoring and control of power transmission and distribution networks. We propose methods and an architecture for data centric communication in power networks; a proof-of-concept decentralized communication infrastructure is presented that uses and advances state of the art peer-to-peer and distributed data management protocols to provide real time access to network state information. We propose methods for adaptive network reconfiguration and self-repair mechanisms to handle fault situations. To efficiently handle complex queries, we present a centralized metadata index, and propose a query language and execution method that allows us to handle high volume data streams in-network

Integrating Wireless Sensor Networks and Mobile Ad-hoc NETworks for enhanced value-added services

In some situations where the standard telecommunication infrastructure is not available, Mobile Ad hoc NETworks (MANETs) can be deployed to provide the required communication. These networks are established "on the fly" without a need for prior communication organization and are composed of autonomous mobile devices, such as cell phones, PDAs or laptops. In similar conditions, such as in emergency response operations, integrating MANETs and Wireless Sensor Networks (WSNs) can notably enhance the MANET participant's end-user experience. WSNs sense and aggregate ambient information, such as physiological, environmental or physical data related to a nearby phenomenon. The integration, which provides end-user availability to WSN required information, is feasible via gateways. However, when the ambient information collected by WSNs is intended for applications residing in MANETs, centralized and fixed gateways are not practicably feasible. This is mainly due to ad-hoc nature, lack of centralized control and constraints on the end-user devices that are used in MANETs. These devices are usually limited in power and capacity and cannot host centralized gateways. In this thesis we exploit the integration of WSN and MANET in order to provide novel value-added services which enhance the end-user experience of MANET participants. Motivating scenarios are introduced, background information is presented, requirements are derived and the state of the art regarding the integration of WSN with existing networks, including MANETs, is evaluated. Based on the evaluation, none of the existing solutions satisfies all of our derived requirements. Therefore, we propose an overall two-level overlay architecture to integrate WSNs (with mobile sinks) and MANETs. This architecture is based on the distributed gateway and applications which form the P2P overlays. Overlays are application-layer networks which are created on top of the exiting MANET. To interconnect gateway and application overlays we derive corresponding requirements and evaluate the existing approaches. Since none of these approaches fulfills all of our requirements, we propose protocols, mechanisms and design corresponding modules for the interconnection of overlays. Finally we refine our overall architecture based on the interconnection aspects. As a proof of concept, we implement a prototype for the inter-overlay information exchange. This implementation is based on SIP extensions and uses two existing P2P middlewares. We also simulate our prototype using Oversim simulation tool and collect experimental results. Based on these results, we can see that our architecture is a valid and promising approach for interconnecting different P2P overlays and can be deployed to provide the overall solution for WSN and MANET integrated system

Pervasive service discovery in low-power and lossy networks

Pervasive Service Discovery (SD) in Low-power and Lossy Networks (LLNs) is expected to play a major role in realising the Internet of Things (IoT) vision. Such a vision aims to expand the current Internet to interconnect billions of miniature smart objects that sense and act on our surroundings in a way that will revolutionise the future. The pervasiveness and heterogeneity of such low-power devices requires robust, automatic, interoperable and scalable deployment and operability solutions. At the same time, the limitations of such constrained devices impose strict challenges regarding complexity, energy consumption, time-efficiency and mobility. This research contributes new lightweight solutions to facilitate automatic deployment and operability of LLNs. It mainly tackles the aforementioned challenges through the proposition of novel component-based, automatic and efficient SD solutions that ensure extensibility and adaptability to various LLN environments. Building upon such architecture, a first fully-distributed, hybrid pushpull SD solution dubbed EADP (Extensible Adaptable Discovery Protocol) is proposed based on the well-known Trickle algorithm. Motivated by EADPs’ achievements, new methods to optimise Trickle are introduced. Such methods allow Trickle to encompass a wide range of algorithms and extend its usage to new application domains. One of the new applications is concretized in the TrickleSD protocol aiming to build automatic, reliable, scalable, and time-efficient SD. To optimise the energy efficiency of TrickleSD, two mechanisms improving broadcast communication in LLNs are proposed. Finally, interoperable standards-based SD in the IoT is demonstrated, and methods combining zero-configuration operations with infrastructure-based solutions are proposed. Experimental evaluations of the above contributions reveal that it is possible to achieve automatic, cost-effective, time-efficient, lightweight, and interoperable SD in LLNs. These achievements open novel perspectives for zero-configuration capabilities in the IoT and promise to bring the ‘things’ to all people everywhere

Using publish/subscribe for message routing in mobile environments

Publish/subscribe is a mature communication paradigm to route and deliver events from publishers to interested subscribers. Initially conceived for large scale systems, e.g., the Internet, it has been used more recently in new scenarios, e.g., wireless sensor networks and the Internet of Things (IoT), where mobility and dynamicity are the norm. The loose-coupling and asynchronicity of publish/subscribe makes it an interesting choice for IoT scenarios, i.e., each node in an IoT network can choose a different role depending on its location, capabilities, etc. This paper presents MFT-PubSub, a fully mobile and fault tolerant content-based publish/subscribe protocol. Our proposal is a purely reactive solution for mobility in a publish/subscribe system without any kind of limits on the mobility patterns of the nodes. A wireless ad hoc network is created without the need of any previous connections or knowledge on the nodes. Handling the mobility, be it physical or logical, of both clients and brokers. We prove the validity of our solution by experimentation, and compare it with AODV, a routing protocol for mobile ad hoc networking. The simulations show an improvement on message delivery rate over previously used protocols.Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature. Research supported by grant TIN2016-79897-P funded by MCIN/AEI/10.13039/501100011033 and by the European Union, and by the Department of Education, Universities and Research of the Basque Government, grant IT-1437-22 (ADIAN)