Self-management for large-scale distributed systems

Autonomic computing aims at making computing systems self-managing by using autonomic managers in order to reduce obstacles caused by management complexity. This thesis presents results of research on self-management for large-scale distributed systems. This research was motivated by the increasing complexity of computing systems and their management. In the first part, we present our platform, called Niche, for programming self-managing component-based distributed applications. In our work on Niche, we have faced and addressed the following four challenges in achieving self-management in a dynamic environment characterized by volatile resources and high churn: resource discovery, robust and efficient sensing and actuation, management bottleneck, and scale. We present results of our research on addressing the above challenges. Niche implements the autonomic computing architecture, proposed by IBM, in a fully decentralized way. Niche supports a network-transparent view of the system architecture simplifying the design of distributed self-management. Niche provides a concise and expressive API for self-management. The implementation of the platform relies on the scalability and robustness of structured overlay networks. We proceed by presenting a methodology for designing the management part of a distributed self-managing application. We define design steps that include partitioning of management functions and orchestration of multiple autonomic managers. In the second part, we discuss robustness of management and data consistency, which are necessary in a distributed system. Dealing with the effect of churn on management increases the complexity of the management logic and thus makes its development time consuming and error prone. We propose the abstraction of Robust Management Elements, which are able to heal themselves under continuous churn. Our approach is based on replicating a management element using finite state machine replication with a reconfigurable replica set. Our algorithm automates the reconfiguration (migration) of the replica set in order to tolerate continuous churn. For data consistency, we propose a majority-based distributed key-value store supporting multiple consistency levels that is based on a peer-to-peer network. The store enables the tradeoff between high availability and data consistency. Using majority allows avoiding potential drawbacks of a master-based consistency control, namely, a single-point of failure and a potential performance bottleneck. In the third part, we investigate self-management for Cloud-based storage systems with the focus on elasticity control using elements of control theory and machine learning. We have conducted research on a number of different designs of an elasticity controller, including a State-Space feedback controller and a controller that combines feedback and feedforward control. We describe our experience in designing an elasticity controller for a Cloud-based key-value store using state-space model that enables to trade-off performance for cost. We describe the steps in designing an elasticity controller. We continue by presenting the design and evaluation of ElastMan, an elasticity controller for Cloud-based elastic key-value stores that combines feedforward and feedback control

Distributed Information Systems and Data Mining in Self-Organizing Networks

The diffusion of sensors and devices to generate and collect data is capillary. The infrastructure that envelops the smart city has to react to the contingent situations and to changes in the operating environment. At the same time, the complexity of a distributed system, consisting of huge amounts of components fixed and mobile, can generate unsustainable costs and latencies to ensure robustness, scalability, and reliability, with type architectures middleware. The distributed system must be able to self-organize and self-restore adapting its operating strategies to optimize the use of resources and overall efficiency. Peer-to-peer systems (P2P) can offer solutions to face the requirements of managing, indexing, searching and analyzing data in scalable and self-organizing fashions, such as in cloud services and big data applications, just to mention two of the most strategic technologies for the next years. In this thesis we present G-Grid, a multi-dimensional distributed data indexing able to efficiently execute arbitrary multi-attribute exact and range queries in decentralized P2P environments. G-Grid is a foundational structure and can be effectively used in a wide range of application environments, including grid computing, cloud and big data domains. Nevertheless we proposed some improvements on the basic structure introducing a bit of randomness by using Small World networks, whereas are structures derived from social networks and show an almost uniform traffic distribution. This produced huge advantages in efficiency, cutting maintenance costs, without losing efficacy. Experiments show how this new hybrid structure obtains the best performance in traffic distribution and it a good settlement for the overall performance on the requirements desired in the modern data systems

A Coordination Model and Framework for Developing Distributed Mobile Applications

How to coordinate multiple devices to work together as a single application is one of the most important challenges for building a distributed mobile application. Mobile devices play important roles in daily life and resolving this challenge is vital. Many coordination models have already been developed to support the implementation of parallel applications, and LIME (Linda In a Mobile Environment) is the most popular member. This thesis evaluates and analyzes the advantages and disadvantages of the LIME, and its predecessor Linda coordination model. This thesis proposes a new coordination model that focuses on overcoming the drawbacks of LIME and Linda. The new coordination model leverages the features of consistent hashing in order to obtain better coordination performance. Additionally, this new coordination model utilizes the idea of replica mechanism to guarantee data integrity. A cross-platform coordination framework, based on the new coordination model, is presented by this thesis in order to facilitate and simplify the development of distributed mobile applications. This framework aims to be robust and high-performance, supporting not only powerful devices such as smartphones but also constrained devices, which includes IoT sensors. The framework utilizes many advanced concepts and technologies such as CoAP protocol, P2P networking, Wi-Fi Direct, and Bluetooth Low Energy to achieve the goals of high-performance and fault-tolerance. Six experiments have been done to test the coordination model and framework from di erent aspects including bandwidth, throughput, packages per second, hit rate, and data distribution. Results of the experiments demonstrate that the proposed coordination model and framework meet the requirements of high-performance and fault-tolerance

An Autonomic Cross-Platform Operating Environment for On-Demand Internet Computing

The Internet has evolved into a global and ubiquitous communication medium interconnecting powerful application servers, diverse desktop computers and mobile notebooks. Along with recent developments in computer technology, such as the convergence of computing and communication devices, the way how people use computers and the Internet has changed people´s working habits and has led to new application scenarios. On the one hand, pervasive computing, ubiquitous computing and nomadic computing become more and more important since different computing devices like PDAs and notebooks may be used concurrently and alternately, e.g. while the user is on the move. On the other hand, the ubiquitous availability and pervasive interconnection of computing systems have fostered various trends towards the dynamic utilization and spontaneous collaboration of available remote computing resources, which are addressed by approaches like utility computing, grid computing, cloud computing and public computing. From a general point of view, the common objective of this development is the use of Internet applications on demand, i.e. applications that are not installed in advance by a platform administrator but are dynamically deployed and run as they are requested by the application user. The heterogeneous and unmanaged nature of the Internet represents a major challenge for the on demand use of custom Internet applications across heterogeneous hardware platforms, operating systems and network environments. Promising remedies are autonomic computing systems that are supposed to maintain themselves without particular user or application intervention. In this thesis, an Autonomic Cross-Platform Operating Environment (ACOE) is presented that supports On Demand Internet Computing (ODIC), such as dynamic application composition and ad hoc execution migration. The approach is based on an integration middleware called crossware that does not replace existing middleware but operates as a self-managing mediator between diverse application requirements and heterogeneous platform configurations. A Java implementation of the Crossware Development Kit (XDK) is presented, followed by the description of the On Demand Internet Computing System (ODIX). The feasibility of the approach is shown by the implementation of an Internet Application Workbench, an Internet Application Factory and an Internet Peer Federation. They illustrate the use of ODIX to support local, remote and distributed ODIC, respectively. Finally, the suitability of the approach is discussed with respect to the support of ODIC

A COGNITIVE ARCHITECTURE FOR AMBIENT INTELLIGENCE

L’Ambient Intelligence (AmI) è caratterizzata dall’uso di sistemi pervasivi per monitorare l’ambiente e modificarlo secondo le esigenze degli utenti e rispettando vincoli definiti globalmente. Questi sistemi non possono prescindere da requisiti come la scalabilità e la trasparenza per l’utente. Una tecnologia che consente di raggiungere questi obiettivi è rappresentata dalle reti di sensori wireless (WSN), caratterizzate da bassi costi e bassa intrusività. Tuttavia, sebbene in grado di effettuare elaborazioni a bordo dei singoli nodi, le WSN non hanno da sole le capacità di elaborazione necessarie a supportare un sistema intelligente; d’altra parte senza questa attività di pre-elaborazione la mole di dati sensoriali può facilmente sopraffare un sistema centralizzato con un’eccessiva quantità di dettagli superflui. Questo lavoro presenta un’architettura cognitiva in grado di percepire e controllare l’ambiente di cui fa parte, basata su un nuovo approccio per l’estrazione di conoscenza a partire dai dati grezzi, attraverso livelli crescenti di astrazione. Le WSN sono utilizzate come strumento sensoriale pervasivo, le cui capacità computazionali vengono utilizzate per pre-elaborare i dati rilevati, in modo da consentire ad un sistema centralizzato intelligente di effettuare ragionamenti di alto livello. L’architettura proposta è stata utilizzata per sviluppare un testbed dotato degli strumenti hardware e software necessari allo sviluppo e alla gestione di applicazioni di AmI basate su WSN, il cui obiettivo principale sia il risparmio energetico. Per fare in modo che le applicazioni di AmI siano in grado di comunicare con il mondo esterno in maniera affidabile, per richiedere servizi ad agenti esterni, l’architettura è stata arricchita con un protocollo di gestione distribuita della reputazione. È stata inoltre sviluppata un’applicazione di esempio che sfrutta le caratteristiche del testbed, con l’obiettivo di controllare la temperatura in un ambiente lavorativo. Quest’applicazione rileva la presenza dell’utente attraverso un modulo per la fusione di dati multi-sensoriali basato su reti bayesiane, e sfrutta questa informazione in un controllore fuzzy multi-obiettivo che controlla gli attuatori sulla base delle preferenze dell’utente e del risparmio energetico.Ambient Intelligence (AmI) systems are characterized by the use of pervasive equipments for monitoring and modifying the environment according to users’ needs, and to globally defined constraints. Furthermore, such systems cannot ignore requirements about ubiquity, scalability, and transparency to the user. An enabling technology capable of accomplishing these goals is represented by Wireless Sensor Networks (WSNs), characterized by low-costs and unintrusiveness. However, although provided of in-network processing capabilities, WSNs do not exhibit processing features able to support comprehensive intelligent systems; on the other hand, without this pre-processing activities the wealth of sensory data may easily overwhelm a centralized AmI system, clogging it with superfluous details. This work proposes a cognitive architecture able to perceive, decide upon, and control the environment of which the system is part, based on a new approach to knowledge extraction from raw data, that addresses this issue at different abstraction levels. WSNs are used as the pervasive sensory tool, and their computational capabilities are exploited to remotely perform preliminary data processing. A central intelligent unit subsequently extracts higher-level concepts in order to carry on symbolic reasoning. The aim of the reasoning is to plan a sequence of actions that will lead the environment to a state as close as possible to the users’ desires, taking into account both implicit and explicit feedbacks from the users, while considering global system-driven goals, such as energy saving. The proposed conceptual architecture was exploited to develop a testbed providing the hardware and software tools for the development and management of AmI applications based on WSNs, whose main goal is energy saving for global sustainability. In order to make the AmI system able to communicate with the external world in a reliable way, when some services are required to external agents, the architecture was enriched with a distributed reputation management protocol. A sample application exploiting the testbed features was implemented for addressing temperature control in a work environment. Knowledge about the user’s presence is obtained through a multi-sensor data fusion module based on Bayesian networks, and this information is exploited by a multi-objective fuzzy controller that operates on actuators taking into account users’ preference and energy consumption constraints

An architectural framework for self-configuration and self-improvement at runtime

[no abstract