Self-management for large-scale distributed systems

Autonomic computing aims at making computing systems self-managing by using autonomic managers in order to reduce obstacles caused by management complexity. This thesis presents results of research on self-management for large-scale distributed systems. This research was motivated by the increasing complexity of computing systems and their management. In the first part, we present our platform, called Niche, for programming self-managing component-based distributed applications. In our work on Niche, we have faced and addressed the following four challenges in achieving self-management in a dynamic environment characterized by volatile resources and high churn: resource discovery, robust and efficient sensing and actuation, management bottleneck, and scale. We present results of our research on addressing the above challenges. Niche implements the autonomic computing architecture, proposed by IBM, in a fully decentralized way. Niche supports a network-transparent view of the system architecture simplifying the design of distributed self-management. Niche provides a concise and expressive API for self-management. The implementation of the platform relies on the scalability and robustness of structured overlay networks. We proceed by presenting a methodology for designing the management part of a distributed self-managing application. We define design steps that include partitioning of management functions and orchestration of multiple autonomic managers. In the second part, we discuss robustness of management and data consistency, which are necessary in a distributed system. Dealing with the effect of churn on management increases the complexity of the management logic and thus makes its development time consuming and error prone. We propose the abstraction of Robust Management Elements, which are able to heal themselves under continuous churn. Our approach is based on replicating a management element using finite state machine replication with a reconfigurable replica set. Our algorithm automates the reconfiguration (migration) of the replica set in order to tolerate continuous churn. For data consistency, we propose a majority-based distributed key-value store supporting multiple consistency levels that is based on a peer-to-peer network. The store enables the tradeoff between high availability and data consistency. Using majority allows avoiding potential drawbacks of a master-based consistency control, namely, a single-point of failure and a potential performance bottleneck. In the third part, we investigate self-management for Cloud-based storage systems with the focus on elasticity control using elements of control theory and machine learning. We have conducted research on a number of different designs of an elasticity controller, including a State-Space feedback controller and a controller that combines feedback and feedforward control. We describe our experience in designing an elasticity controller for a Cloud-based key-value store using state-space model that enables to trade-off performance for cost. We describe the steps in designing an elasticity controller. We continue by presenting the design and evaluation of ElastMan, an elasticity controller for Cloud-based elastic key-value stores that combines feedforward and feedback control

Prototyping a peer-to-peer session initiation protocol user agent

The Session Initiation Protocol (SIP) has in recent years become a popular protocol for the exchange of text, voice and video over IP networks. This thesis proposes the use of a class of structured peer to peer protocols - commonly known as Distributed Hash Tables (DHTs) - to provide a SIP overlay with services such as end-point location management and message relay, in the absence of traditional, centralised resources such as SIP proxies and registrars. A peer-to-peer layer named OverCord, which allows the interaction with any specific DHT protocol via the use of appropriate plug-ins, was designed, implemented and tested. This layer was then incorporated into a SIP user agent distributed by NIST (National Institute of Standards and Technology, USA). The modified user agent is capable of reliably establishing text, audio and video communication with similarly modified agents (peers) as well as conventional, centralized SIP overlays

Dynamic data placement and discovery in wide-area networks

The workloads of online services and applications such as social networks, sensor data platforms and web search engines have become increasingly global and dynamic, setting new challenges to providing users with low latency access to data. To achieve this, these services typically leverage a multi-site wide-area networked infrastructure. Data access latency in such an infrastructure depends on the network paths between users and data, which is determined by the data placement and discovery strategies. Current strategies are static, which offer low latencies upon deployment but worse performance under a dynamic workload. We propose dynamic data placement and discovery strategies for wide-area networked infrastructures, which adapt to the data access workload. We achieve this with data activity correlation (DAC), an application-agnostic approach for determining the correlations between data items based on access pattern similarities. By dynamically clustering data according to DAC, network traffic in clusters is kept local. We utilise DAC as a key component in reducing access latencies for two application scenarios, emphasising different aspects of the problem: The first scenario assumes the fixed placement of data at sites, and thus focusses on data discovery. This is the case for a global sensor discovery platform, which aims to provide low latency discovery of sensor metadata. We present a self-organising hierarchical infrastructure consisting of multiple DAC clusters, maintained with an online and distributed split-and-merge algorithm. This reduces the number of sites visited, and thus latency, during discovery for a variety of workloads. The second scenario focusses on data placement. This is the case for global online services that leverage a multi-data centre deployment to provide users with low latency access to data. We present a geo-dynamic partitioning middleware, which maintains DAC clusters with an online elastic partition algorithm. It supports the geo-aware placement of partitions across data centres according to the workload. This provides globally distributed users with low latency access to data for static and dynamic workloads.Open Acces

Incrementando as redes centradas à informaçãopara uma internet das coisas baseada em nomes

The way we use the Internet has been evolving since its origins. Nowadays, users are more interested in accessing contents and services with high demands in terms of bandwidth, security and mobility. This evolution has triggered the emergence of novel networking architectures targeting current, as well as future, utilisation demands. Information-Centric Networking (ICN) is a prominent example of these novel architectures that moves away from the current host-centric communications and centres its networking functions around content. Parallel to this, new utilisation scenarios in which smart devices interact with one another, as well as with other networked elements, have emerged to constitute what we know as the Internet of Things (IoT). IoT is expected to have a significant impact on both the economy and society. However, fostering the widespread adoption of IoT requires many challenges to be overcome. Despite recent developments, several issues concerning the deployment of IPbased IoT solutions on a large scale are still open. The fact that IoT is focused on data and information rather than on point-topoint communications suggests the adoption of solutions relying on ICN architectures. In this context, this work explores the ground concepts of ICN to develop a comprehensive vision of the principal requirements that should be met by an IoT-oriented ICN architecture. This vision is complemented with solutions to fundamental issues for the adoption of an ICN-based IoT. First, to ensure the freshness of the information while retaining the advantages of ICN’s in-network caching mechanisms. Second, to enable discovery functionalities in both local and large-scale domains. The proposed mechanisms are evaluated through both simulation and prototyping approaches, with results showcasing the feasibility of their adoption. Moreover, the outcomes of this work contribute to the development of new compelling concepts towards a full-fledged Named Network of Things.A forma como usamos a Internet tem vindo a evoluir desde a sua criação. Atualmente, os utilizadores estão mais interessados em aceder a conteúdos e serviços, com elevados requisitos em termos de largura de banda, segurança e mobilidade. Esta evolução desencadeou o desenvolvimento de novas arquiteturas de rede, visando os atuais, bem como os futuros, requisitos de utilização. As Redes Centradas à Informação (Information-Centric Networking - ICN) são um exemplo proeminente destas novas arquiteturas que, em vez de seguirem um modelo de comunicação centrado nos dispositivos terminais, centram as suas funções de rede em torno do próprio conteúdo. Paralelamente, novos cenários de utilização onde dispositivos inteligentes interagem entre si, e com outros elementos de rede, têm vindo a aparecer e constituem o que hoje conhecemos como a Internet das Coisas (Internet of Things - IoT ). É esperado que a IoT tenha um impacto significativo na economia e na sociedade. No entanto, promover a adoção em massa da IoT ainda requer que muitos desafios sejam superados. Apesar dos desenvolvimentos recentes, vários problemas relacionados com a adoção em larga escala de soluções de IoT baseadas no protocolo IP estão em aberto. O facto da IoT estar focada em dados e informação, em vez de comunicações ponto-a-ponto, sugere a adoção de soluções baseadas em arquiteturas ICN. Neste sentido, este trabalho explora os conceitos base destas soluções para desenvolver uma visão completa dos principais requisitos que devem ser satisfeitos por uma solução IoT baseada na arquitetura de rede ICN. Esta visão é complementada com soluções para problemas cruciais para a adoção de uma IoT baseada em ICN. Em primeiro lugar, assegurar que a informação seja atualizada e, ao mesmo tempo, manter as vantagens do armazenamento intrínseco em elementos de rede das arquiteturas ICN. Em segundo lugar, permitir as funcionalidades de descoberta não só em domínios locais, mas também em domínios de larga-escala. Os mecanismos propostos são avaliados através de simulações e prototipagem, com os resultados a demonstrarem a viabilidade da sua adoção. Para além disso, os resultados deste trabalho contribuem para o desenvolvimento de conceitos sólidos em direção a uma verdadeira Internet das Coisas baseada em Nomes.Programa Doutoral em Telecomunicaçõe

Practical Aggregation in the Edge

Due to the increasing amounts of data produced by applications and devices, cloud infrastructures are becoming unable to timely process and provide answers back to users. This has led to the emergence of the edge computing paradigm that aims at moving computations closer to end user devices. Edge computing can be defined as performing computations outside the boundaries of cloud data centres. This however, can be materialised across very different scenarios considering the broad spectrum of devices that can be leveraged to perform computations in the edge. In this thesis, we focus on a concrete scenario of edge computing, that of multiple devices with wireless capabilities that collectively form a wireless ad hoc network to perform distributed computations. We aim at devising practical solutions for these scenarios however, there is a lack of tools to help us in achieving such goal. To address this first limitation we propose a novel framework, called Yggdrasil, that is specifically tailored to develop and execute distributed protocols over wireless ad hoc networks on commodity devices. As to enable distributed computations in such networks, we focus on the particular case of distributed data aggregation. In particular, we address a harder variant of this problem, that we dub distributed continuous aggregation, where input values used for the computation of the aggregation function may change over time, and propose a novel distributed continuous aggregation protocol, called MiRAge. We have implemented and validated both Yggdrasil and MiRAge through an extensive experimental evaluation using a test-bed composed of 24 Raspberry Pi’s. Our results show that Yggdrasil provides adequate abstractions and tools to implement and execute distributed protocols in wireless ad hoc settings. Our evaluation is also composed of a practical comparative study on distributed continuous aggregation protocols, that shows that MiRAge is more robust and achieves more precise aggregation results than competing state-of-the-art alternatives

Distributed D3: A web-based distributed data visualisation framework for Big Data

The influx of Big Data has created an ever-growing need for analytic tools targeting towards the acquisition of insights and knowledge from large datasets. Visual perception as a fundamental tool used by humans to retrieve information from the outside world around us has its unique ability to distinguish patterns pre-attentively. Visual analytics via data visualisations is therefore a very powerful tool and has become ever more important in this era. Data-Driven Documents (D3.js) is a versatile and popular web-based data visualisation library that has tended to be the standard toolkit for visualising data in recent years. However, the library is technically inherent and limited in capability by the single thread model of a single browser window in a single machine, and therefore not able to deal with large datasets. The main objective of this thesis is to overcome this limitation and address possible challenges by developing the Distributed D3 framework that employs distributed mechanism to enable the possibility of delivering web-based visualisations for large-scale data, which also allows to effectively utilise the graphical computational resources of the modern visualisation environments. As a result, the first contribution is that the integrated version of Distributed D3 framework has been developed for the Data Observatory. The work proves the concept of Distributed D3 is feasible in reality and also enables developers to collaborate on large-scale data visualisations by using it on the Data Observatory. The second contribution is that the Distributed D3 has been optimised by investigating the potential bottlenecks for large-scale data visualisation applications. The work finds the key performance bottlenecks of the framework and shows an improvement of the overall performance by 35.7% after optimisations, which improves the scalability and usability of Distributed D3 for large-scale data visualisation applications. The third contribution is that the generic version of Distributed D3 framework has been developed for the customised environments. The work improves the usability and flexibility of the framework and makes it ready to be published in the open-source community for further improvements and usages.Open Acces

A framework for the dynamic management of Peer-to-Peer overlays

Peer-to-Peer (P2P) applications have been associated with inefficient operation, interference with other network services and large operational costs for network providers. This thesis presents a framework which can help ISPs address these issues by means of intelligent management of peer behaviour. The proposed approach involves limited control of P2P overlays without interfering with the fundamental characteristics of peer autonomy and decentralised operation. At the core of the management framework lays the Active Virtual Peer (AVP). Essentially intelligent peers operated by the network providers, the AVPs interact with the overlay from within, minimising redundant or inefficient traffic, enhancing overlay stability and facilitating the efficient and balanced use of available peer and network resources. They offer an “insider‟s” view of the overlay and permit the management of P2P functions in a compatible and non-intrusive manner. AVPs can support multiple P2P protocols and coordinate to perform functions collectively. To account for the multi-faceted nature of P2P applications and allow the incorporation of modern techniques and protocols as they appear, the framework is based on a modular architecture. Core modules for overlay control and transit traffic minimisation are presented. Towards the latter, a number of suitable P2P content caching strategies are proposed. Using a purpose-built P2P network simulator and small-scale experiments, it is demonstrated that the introduction of AVPs inside the network can significantly reduce inter-AS traffic, minimise costly multi-hop flows, increase overlay stability and load-balancing and offer improved peer transfer performance