Enabling technologies for decentralized interpersonal communication

In the recent years the Internet users have witnessed the emergence of Peer-to-Peer (P2P) technologies and applications. One class of P2P applications is comprised of applications that are targeted for interpersonal communication. The communication applications that utilize P2P technologies are referred to as decentralized interpersonal communication applications. Such applications are decentralized in a sense that they do not require assistance from centralized servers for setting up multimedia sessions between users. The invention of Distributed Hash Table (DHT) algorithms has been an important, but not an inclusive enabler for decentralized interpersonal communication. Even though the DHTs provide a basic foundation for decentralization, there are still a number of challenges without viable technological solutions. The main contribution of this thesis is to propose technological solutions to a subset of the existing challenges. In addition, this thesis also presents the preliminary work for the technological solutions. There are two parts in the preliminary work. In the first part, a set of DHT algorithms are evaluated from the viewpoint of decentralized interpersonal communication, and the second part gives a coherent presentation of the challenges that a decentralized interpersonal communication application is going to encounter in mobile networks. The technological solution proposals contain two architectures and two algorithms. The first architecture enables an interconnection between a decentralized and a centralized communication network, and the second architecture enables the decentralization of a set of legacy applications. The first algorithm is a load balancing algorithm that enables good scalability, and the second algorithm is a search algorithm that enables arbitrary searches. The algorithms can be used, for example, in DHT-based networks. Even though this thesis has focused on the decentralized interpersonal communication, some of the proposed technological solutions also have general applicability outside the scope of decentralized interpersonal communication

Enabling technologies for decentralized interpersonal communication

In the recent years the Internet users have witnessed the emergence of Peer-to-Peer (P2P) technologies and applications. One class of P2P applications is comprised of applications that are targeted for interpersonal communication. The communication applications that utilize P2P technologies are referred to as decentralized interpersonal communication applications. Such applications are decentralized in a sense that they do not require assistance from centralized servers for setting up multimedia sessions between users. The invention of Distributed Hash Table (DHT) algorithms has been an important, but not an inclusive enabler for decentralized interpersonal communication. Even though the DHTs provide a basic foundation for decentralization, there are still a number of challenges without viable technological solutions. The main contribution of this thesis is to propose technological solutions to a subset of the existing challenges. In addition, this thesis also presents the preliminary work for the technological solutions. There are two parts in the preliminary work. In the first part, a set of DHT algorithms are evaluated from the viewpoint of decentralized interpersonal communication, and the second part gives a coherent presentation of the challenges that a decentralized interpersonal communication application is going to encounter in mobile networks. The technological solution proposals contain two architectures and two algorithms. The first architecture enables an interconnection between a decentralized and a centralized communication network, and the second architecture enables the decentralization of a set of legacy applications. The first algorithm is a load balancing algorithm that enables good scalability, and the second algorithm is a search algorithm that enables arbitrary searches. The algorithms can be used, for example, in DHT-based networks. Even though this thesis has focused on the decentralized interpersonal communication, some of the proposed technological solutions also have general applicability outside the scope of decentralized interpersonal communication

Scalable adaptive group communication on bi-directional shared prefix trees

Efficient group communication within the Internet has been implemented by multicast. Unfortunately, its global deployment is missing. Nevertheless, emerging and progressively establishing popular applications, like IPTV or large-scale social video chats, require an economical data distribution throughout the Internet. To overcome the limitations of multicast deployment, we introduce and analyze BIDIR-SAM, the rest structured overlay multicast scheme based on bi-directional shared prefix trees. BIDIR-SAM admits predictable costs growing logarithmically with increasing group size. We also present a broadcast approach for DHT-enabled P2P networks. Both schemes are integrated in a standard compliant hybrid group communication architecture, bridging the gap between overlay and underlay as well as between inter- and intra-domain multicast

Data Storage and Dissemination in Pervasive Edge Computing Environments

Nowadays, smart mobile devices generate huge amounts of data in all sorts of gatherings. Much of that data has localized and ephemeral interest, but can be of great use if shared among co-located devices. However, mobile devices often experience poor connectivity, leading to availability issues if application storage and logic are fully delegated to a remote cloud infrastructure. In turn, the edge computing paradigm pushes computations and storage beyond the data center, closer to end-user devices where data is generated and consumed. Hence, enabling the execution of certain components of edge-enabled systems directly and cooperatively on edge devices. This thesis focuses on the design and evaluation of resilient and efficient data storage and dissemination solutions for pervasive edge computing environments, operating with or without access to the network infrastructure. In line with this dichotomy, our goal can be divided into two specific scenarios. The first one is related to the absence of network infrastructure and the provision of a transient data storage and dissemination system for networks of co-located mobile devices. The second one relates with the existence of network infrastructure access and the corresponding edge computing capabilities. First, the thesis presents time-aware reactive storage (TARS), a reactive data storage and dissemination model with intrinsic time-awareness, that exploits synergies between the storage substrate and the publish/subscribe paradigm, and allows queries within a specific time scope. Next, it describes in more detail: i) Thyme, a data storage and dis- semination system for wireless edge environments, implementing TARS; ii) Parsley, a flexible and resilient group-based distributed hash table with preemptive peer relocation and a dynamic data sharding mechanism; and iii) Thyme GardenBed, a framework for data storage and dissemination across multi-region edge networks, that makes use of both device-to-device and edge interactions. The developed solutions present low overheads, while providing adequate response times for interactive usage and low energy consumption, proving to be practical in a variety of situations. They also display good load balancing and fault tolerance properties.Resumo Hoje em dia, os dispositivos móveis inteligentes geram grandes quantidades de dados em todos os tipos de aglomerações de pessoas. Muitos desses dados têm interesse loca- lizado e efêmero, mas podem ser de grande utilidade se partilhados entre dispositivos co-localizados. No entanto, os dispositivos móveis muitas vezes experienciam fraca co- nectividade, levando a problemas de disponibilidade se o armazenamento e a lógica das aplicações forem totalmente delegados numa infraestrutura remota na nuvem. Por sua vez, o paradigma de computação na periferia da rede leva as computações e o armazena- mento para além dos centros de dados, para mais perto dos dispositivos dos utilizadores finais onde os dados são gerados e consumidos. Assim, permitindo a execução de certos componentes de sistemas direta e cooperativamente em dispositivos na periferia da rede. Esta tese foca-se no desenho e avaliação de soluções resilientes e eficientes para arma- zenamento e disseminação de dados em ambientes pervasivos de computação na periferia da rede, operando com ou sem acesso à infraestrutura de rede. Em linha com esta dico- tomia, o nosso objetivo pode ser dividido em dois cenários específicos. O primeiro está relacionado com a ausência de infraestrutura de rede e o fornecimento de um sistema efêmero de armazenamento e disseminação de dados para redes de dispositivos móveis co-localizados. O segundo diz respeito à existência de acesso à infraestrutura de rede e aos recursos de computação na periferia da rede correspondentes. Primeiramente, a tese apresenta armazenamento reativo ciente do tempo (ARCT), um modelo reativo de armazenamento e disseminação de dados com percepção intrínseca do tempo, que explora sinergias entre o substrato de armazenamento e o paradigma pu- blicação/subscrição, e permite consultas num escopo de tempo específico. De seguida, descreve em mais detalhe: i) Thyme, um sistema de armazenamento e disseminação de dados para ambientes sem fios na periferia da rede, que implementa ARCT; ii) Pars- ley, uma tabela de dispersão distribuída flexível e resiliente baseada em grupos, com realocação preventiva de nós e um mecanismo de particionamento dinâmico de dados; e iii) Thyme GardenBed, um sistema para armazenamento e disseminação de dados em redes multi-regionais na periferia da rede, que faz uso de interações entre dispositivos e com a periferia da rede. As soluções desenvolvidas apresentam baixos custos, proporcionando tempos de res- posta adequados para uso interativo e baixo consumo de energia, demonstrando serem práticas nas mais diversas situações. Estas soluções também exibem boas propriedades de balanceamento de carga e tolerância a faltas

Accurate Player Modeling and Cheat-Proof Gameplay in Peer-to-Peer Based Multiplayer Online Games

We present the first detailed measurement study and models of the virtual populations in popular Massively Multiplayer Online Role-Playing Games (MMORPGs). Our results show that, amongst several MMORPGs with very different play styles, the patterns of behaviors are consistent and can be described using a common set of models. In addition, we break down actions common to Trading Card Games (TCGs) and explain how they can be executed between players without the need for a third party referee. In each action, the player is either prevented from cheating, or if they do cheat, the opponent will be able to prove they have done so. We show these methods are secure and may be used in many various styles of TCGs. We measure moves in a real TCG to compare to our implementation of Match+Guardian (M+G), our secure Peer-to-Peer (P2P) protocol for implementing online TCGs. Our results, based on an evaluation of M+G\u27s performance on the Android (TM) platform, show that M+G can be used in a P2P fashion on mobile devices. Finally, we introduce and outline a HYbrid P2P ARchitecture for Trading Card Games, HYPAR-TCG. The system utilizes Distributed Hash Tables (DHTs) and other P2P overlays to store cached game data and to perform game matchmaking. This helps reduce the network and computational load to the central servers. We describe how a centralized server authority can work in concert with a P2P gameplay protocol, while still allowing for reputation and authoritative account management

Reorganization in network regions for optimality and fairness

Thesis (S.M.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2004.Includes bibliographical references (p. 92-95).(cont.) down implicit assumptions of altruism while showing the resulting negative impact on utility. From a selfish equilibrium, with much lower global utility, we show the ability of our algorithm to reorganize and restore the utility of individual nodes, and the system as a whole, to similar levels as realized in the SuperPeer network. Simulation of our algorithm shows that it reaches the predicted optimal utility while providing fairness not realized in other systems. Further analysis includes an epsilon equilibrium model where we attempt to more accurately represent the actual reward function of nodes. We find that by employing such a model, over 60% of the nodes are connected. In addition, this model converges to a utility 34% greater than achieved in the SuperPeer network while making no assumptions on the benevolence of nodes or centralized organization.This thesis proposes a reorganization algorithm, based on the region abstraction, to exploit the natural structure in overlays that stems from common interests. Nodes selfishly adapt their connectivity within the overlay in a distributed fashion such that the topology evolves to clusters of users with shared interests. Our architecture leverages the inherent heterogeneity of users and places within the system their incentives and ability to affect the network. As such, it is not dependent on the altruism of any other nodes in the system. Of particular interest is the optimality and fairness of our design. We rigorously define ideal and fair networks and develop a continuum of optimality measures by which to evaluate our algorithm. Further, to evaluate our algorithm within a realistic context, validate assumptions and make design decisions, we capture data from a portion of a live file-sharing network. More importantly, we discover, name, quantify and solve several previously unrecognized subtle problems in a content-based self-organizing network as a direct result of simulations using the trace data. We motivate our design by examining the dependence of existing systems on benevolent Super-Peers. Through simulation we find that the current architecture is highly dependent on the filtering capability and the willingness of the SuperPeer network to absorb the majority of the query burden. The remainder of the thesis is devoted to a world in which SuperPeers no longer exist or are untenable. In our evaluation, we introduce four reasons for utility suboptimal self-reorganizing networks: anarchy (selfish behavior), indifference, myopia and ordering. We simulate the level of utility and happiness achieved in existing architectures. Then we systematically tearby Robert E. Beverly, IV.S.M

Distributed Search in Semantic Web Service Discovery

This thesis presents a framework for semantic Web Service discovery using descriptive (non-functional) service characteristics in a large-scale, multi-domain setting. The framework uses Web Ontology Language for Services (OWL-S) to design a template for describing non-functional service parameters in a way that facilitates service discovery, and presents a layered scheme for organizing ontologies used in service description. This service description scheme serves as a core for desigining the four main functions of a service directory: a template-based user interface, semantic query expansion algorithms, a two-level indexing scheme that combines Bloom filters with a Distributed Hash Table, and a distributed approach for storing service description. The service directory is, in turn, implemented as an extension of the Open Service Discovery Architecture. The search algorithms presented in this thesis are designed to maximize precision and completeness of service discovery, while the distributed design of the directory allows individual administrative domains to retain a high degree of independence and maintain access control to information about their services

Scalable and Distributed Resource Management Protocols for Cloud and Big Data Clusters

Cloud data centers require an operating system to manage resources and satisfy operational requirements and management objectives. The growth of popularity in cloud services causes the appearance of a new spectrum of services with sophisticated workload and resource management requirements. Also, data centers are growing by addition of various type of hardware to accommodate the ever-increasing requests of users. Nowadays a large percentage of cloud resources are executing data-intensive applications which need continuously changing workload fluctuations and specific resource management. To this end, cluster computing frameworks are shifting towards distributed resource management for better scalability and faster decision making. Such systems benefit from the parallelization of control and are resilient to failures. Throughout this thesis we investigate algorithms, protocols and techniques to address these challenges in large-scale data centers. We introduce a distributed resource management framework which consolidates virtual machine to as few servers as possible to reduce the energy consumption of data center and hence decrease the cost of cloud providers. This framework can characterize the workload of virtual machines and hence handle trade-off energy consumption and Service Level Agreement (SLA) of customers efficiently. The algorithm is highly scalable and requires low maintenance cost with dynamic workloads and it tries to minimize virtual machines migration costs. We also introduce a scalable and distributed probe-based scheduling algorithm for Big data analytics frameworks. This algorithm can efficiently address the problem job heterogeneity in workloads that has appeared after increasing the level of parallelism in jobs. The algorithm is massively scalable and can reduce significantly average job completion times in comparison with the-state of-the-art. Finally, we propose a probabilistic fault-tolerance technique as part of the scheduling algorithm

A Semantic Index for Linked Open Data and Big Data Applications

This work proposes a new approach to index multidimensional data based on kd-trees and proposes also a novel approach to query processing. The indexing data structure is distributed across a network of "peers", where each one hosts a part of the tree and uses message passing for communication among nodes. The advantages of this kind of approach are mainly two: it is possible to i) handle a larger number of nodes and points than a single peer based architecture and ii) to run in an efficient way the elaboration of multiple queries. In particular, we propose a novel version of the k-nearest neighbor algorithm that is able to start a query in a randomly chosen peer. Furthrmore, it returns the results without traverse the peer containing the root. Preliminary experiments demonstrated that on average in about 65% of cases a query starting in a random node, does not involve the peer containing the root of the tree. Also, on average in about 98% of cases, it returns the results without involving the root peer. This work also proposes an approach to cope with textual data and provides a way to perform semantic query over the text

Network coding for sensor networks, distributed storage and video streaming

The classical store-and-forward routing has and will continue to be the most important routing architecture in many modern packet-switched communication networks. In a packet-switched network, data is sent in the form of discrete packets that traverse hop-by-hop from a source to a destination. At each intermediate hop, the router stores and examines the packets it receives then forwards them to the next hop until they reach the correct destinations according to some pre-defined routing algorithms. Importantly, the intermediate routers do not modify but simply store and forward the contents of the packets. In contrast, a new generalized approach to routing called Network Coding (NC) allows the intermediate routers to modify and combine packets from different sources and destinations in such a way that increases the overall throughput. The core idea of NC allowing the intermediate nodes in a network to perform data processing has a wide range of applications well beyond its initial application to routing, impacting different disciplines from distributed data storage and security to energy efficient sensor networks and Internet media streaming. To that end, this dissertation aims to develop the theories and applications of NC via four main thrusts: 1) Energy efficient NC techniques for sensor networks, 2) Novel NC techniques and protocols for Internet video streaming, 3) Stochastic data replenishment for large scale NC-based distributed storage systems, 4) Real-world implementation of NC-based distributed video streaming system. In thrust one, we describe a novel cross-sensor coding technique that combines network topology and coding techniques to maximize the life-time of a sensor network, by addressing the uneven energy consumption problem in data gathering sensor networks where the nodes closer to the sink tend to consume more energy than those of the farther nodes. Our approach is based on the following observation from the sensor networks using On-Off Keying and digital transmission: transmitting bit "1" consumes much more energy than bit "0". Our proposed coding technique exploits this difference to reduce the communication energy by limiting the number of bits "1" in the output codeword (low-weight codeword) and to use NC-based cross-sensor coding technique to equalize the communication energy among the nodes. This cross-sensor coding scheme can significantly extend the network lifetime as compared with traditional (binary) coding by solving the energy-consumption unfairness problem. The theoretical and experimental results confirm that transmission energy can be reduced substantially (e.g., a factor of 15) and the unequal energy consumption among nodes can be practically eliminated. In thrust two, we describe a rate distortion aware hierarchical NC technique and transport protocol for Internet video streaming. We begin by proposing a NC-based multi-sender streaming framework that reduces the overall storage, eliminates the complexity of sender synchronization, and enables TCP streaming. Furthermore, we propose a Hierarchical Network Coding (HNC) technique that facilitates scalable video streaming to combat bandwidth fluctuation on the Internet. This HNC technique enables receiver to recover the important data gracefully in the presence of limited bandwidth which causes an increase in decoding delay. Simulations demonstrate that under certain scenarios, our proposed NC techniques can result in bandwidth saving up to 60% over the traditional schemes. In thrust three, we present a theory of NC-based data replenishment to automate the process of data maintenance for large scale distributed storage systems. The data replenishment mechanism is the core of these systems that promises to reduce the coordination complexity and increases performance scalability. The data replenishment automates the process of maintaining a sufficient level of data redundancy to ensure the availability of data in presence of peer departures and failures. The dynamics of peers entering and leaving the network is modeled as a stochastic process. We propose a novel analytical time-backward technique to bound the expected time, the longer the better, for a piece of data to remain in P2P systems. Both theoretical and simulation results are in agreement, indicating that our proposed data replenishment via random linear network coding (RLNC) outperforms other popular strategies that employ repetition and channel coding techniques. Specifically, we show that the expected time for a piece of data to remain in a P2P system is exponential in the number of peers used to store the data for the RLNC-based strategy, while they are quadratic for other strategies. Furthermore, the time-backward technique can be applied to problems in other disciplines such as gene population modeling in theoretical biology. Finally in thrust four, we present the architecture, design, and experimental results of an actual NC-based distributed video streaming system. We first implement random linear network coding (RLNC) library and show the feasibility of using RLNC in P2P video streaming applications. Then we design, implement and analyze RESnc - a resilient P2P video storage and streaming over the Internet using network coding. RESnc increases the streaming throughput and data resiliency against peer departures and failures using peer diversity. These improvements are based on three architectural elements: 1) The RLNC scheme that breaks a video stream into multiple smaller pieces, codes, and disperses them throughout peers in the network, in such a way to maximize the probability of recovering the original video under peer departures and failures; 2) The scalable mechanism for automating the data replenishment process using RLNC to maintain a sufficient level of redundancy for video stored in the system; 3) The path-diversity streaming protocol for a client to simultaneously stream a video from multiple peers with minimal coordination. Experimental results demonstrated that our system adapts well with bandwidth fluctuation, provides significant playback quality improvement and bandwidth saving