A framework for the dynamic management of Peer-to-Peer overlays

Peer-to-Peer (P2P) applications have been associated with inefficient operation, interference with other network services and large operational costs for network providers. This thesis presents a framework which can help ISPs address these issues by means of intelligent management of peer behaviour. The proposed approach involves limited control of P2P overlays without interfering with the fundamental characteristics of peer autonomy and decentralised operation. At the core of the management framework lays the Active Virtual Peer (AVP). Essentially intelligent peers operated by the network providers, the AVPs interact with the overlay from within, minimising redundant or inefficient traffic, enhancing overlay stability and facilitating the efficient and balanced use of available peer and network resources. They offer an “insider‟s” view of the overlay and permit the management of P2P functions in a compatible and non-intrusive manner. AVPs can support multiple P2P protocols and coordinate to perform functions collectively. To account for the multi-faceted nature of P2P applications and allow the incorporation of modern techniques and protocols as they appear, the framework is based on a modular architecture. Core modules for overlay control and transit traffic minimisation are presented. Towards the latter, a number of suitable P2P content caching strategies are proposed. Using a purpose-built P2P network simulator and small-scale experiments, it is demonstrated that the introduction of AVPs inside the network can significantly reduce inter-AS traffic, minimise costly multi-hop flows, increase overlay stability and load-balancing and offer improved peer transfer performance

Enabling technologies for decentralized interpersonal communication

In the recent years the Internet users have witnessed the emergence of Peer-to-Peer (P2P) technologies and applications. One class of P2P applications is comprised of applications that are targeted for interpersonal communication. The communication applications that utilize P2P technologies are referred to as decentralized interpersonal communication applications. Such applications are decentralized in a sense that they do not require assistance from centralized servers for setting up multimedia sessions between users. The invention of Distributed Hash Table (DHT) algorithms has been an important, but not an inclusive enabler for decentralized interpersonal communication. Even though the DHTs provide a basic foundation for decentralization, there are still a number of challenges without viable technological solutions. The main contribution of this thesis is to propose technological solutions to a subset of the existing challenges. In addition, this thesis also presents the preliminary work for the technological solutions. There are two parts in the preliminary work. In the first part, a set of DHT algorithms are evaluated from the viewpoint of decentralized interpersonal communication, and the second part gives a coherent presentation of the challenges that a decentralized interpersonal communication application is going to encounter in mobile networks. The technological solution proposals contain two architectures and two algorithms. The first architecture enables an interconnection between a decentralized and a centralized communication network, and the second architecture enables the decentralization of a set of legacy applications. The first algorithm is a load balancing algorithm that enables good scalability, and the second algorithm is a search algorithm that enables arbitrary searches. The algorithms can be used, for example, in DHT-based networks. Even though this thesis has focused on the decentralized interpersonal communication, some of the proposed technological solutions also have general applicability outside the scope of decentralized interpersonal communication

Enabling technologies for decentralized interpersonal communication

In the recent years the Internet users have witnessed the emergence of Peer-to-Peer (P2P) technologies and applications. One class of P2P applications is comprised of applications that are targeted for interpersonal communication. The communication applications that utilize P2P technologies are referred to as decentralized interpersonal communication applications. Such applications are decentralized in a sense that they do not require assistance from centralized servers for setting up multimedia sessions between users. The invention of Distributed Hash Table (DHT) algorithms has been an important, but not an inclusive enabler for decentralized interpersonal communication. Even though the DHTs provide a basic foundation for decentralization, there are still a number of challenges without viable technological solutions. The main contribution of this thesis is to propose technological solutions to a subset of the existing challenges. In addition, this thesis also presents the preliminary work for the technological solutions. There are two parts in the preliminary work. In the first part, a set of DHT algorithms are evaluated from the viewpoint of decentralized interpersonal communication, and the second part gives a coherent presentation of the challenges that a decentralized interpersonal communication application is going to encounter in mobile networks. The technological solution proposals contain two architectures and two algorithms. The first architecture enables an interconnection between a decentralized and a centralized communication network, and the second architecture enables the decentralization of a set of legacy applications. The first algorithm is a load balancing algorithm that enables good scalability, and the second algorithm is a search algorithm that enables arbitrary searches. The algorithms can be used, for example, in DHT-based networks. Even though this thesis has focused on the decentralized interpersonal communication, some of the proposed technological solutions also have general applicability outside the scope of decentralized interpersonal communication

A performance evaluation of peer-to-peer storage systems

This work evaluates the performance of Peer-to-Peer storage systems in structured Peer-to-Peer (P2P) networks under the impacts of a continuous process of nodes joining and leaving the network (Churn). Based on the Distributed Hash Tables (DHT), the peer-to-peer systems provide the means to store data among a large and dynamic set of participating host nodes. We consider the fact that existing solutions do not tolerate a high Churn rate or are not really scalable in terms of number of stored data blocks. The considered performance metrics include number of data blocks lost, bandwidth consumption, latencies and distance of matched lookups. We have selected Pastry, Chord and Kademlia to evaluate the e ect of inopportune connections/disconnections in Peer-to-Peer storage systems, because these selected P2P networks possess distinctive characteristics. Chord is one of the rst structured P2P networks that implements Distributed Hash Tables (DHTs). Similar to Chord, Pastry is based on a ring structure, with the identi er space forming the ring. However, Pastry uses a di erent algorithm than Chord to select the overlay neighbors of a peer. Kademlia is a more recent structured P2P network, with the XOR mechanism for improving distance calculation. DHT deployments are characterized by Churn. But if the frequency of Churn is too high, data blocks can be lost and lookup mechanism begin to incur delays. In architectures that employ DHTs, the choice of algorithm for data replication and maintenance can have a signi cant impact on the performance and reliability. PAST is a persistent Peer-to-Peer storage utility, which replicates complete les on multiple nodes, and uses Pastry for message routing and content location. The hypothesis is that by enhancing the Churn tolerance through building a really e cient replication and maintenance mechanisms, it will: i) Operate better than a peer-to-peer storage system such as PAST especially in replica placement strategy with a fewer data transfers. ii) Resolve le lookups with a match that is closer to the source peer, thus con- serving bandwidth. Our research will involve a series of simulation studies using two network simulators OverSim and OMNeT++. The main results are: Our approach achieves a higher data availability in presence of Churn, than the original PAST replication strategy; For a Churn occuring every minute our strategy loses two times less blocks than PAST; Our replication strategy induces an average of twice less block transfers than PAST

Efficient and Flexible Search in Large Scale Distributed Systems

Peer-to-peer (P2P) technology has triggered a wide range of distributed systems beyond simple file-sharing. Distributed XML databases, distributed computing, server-less web publishing and networked resource/service sharing are only a few to name. Despite of the diversity in applications, these systems share a common problem regarding searching and discovery of information. This commonality stems from the transitory nodes population and volatile information content in the participating nodes. In such dynamic environment, users are not expected to have the exact information about the available objects in the system. Rather queries are based on partial information, which requires the search mechanism to be flexible. On the other hand, to scale with network size the search mechanism is required to be bandwidth efficient. Since the advent of P2P technology experts from industry and academia have proposed a number of search techniques - none of which is able to provide satisfactory solution to the conflicting requirements of search efficiency and flexibility. Structured search techniques, mostly Distributed Hash Table (DHT)-based, are bandwidth efficient while semi(un)-structured techniques are flexible. But, neither achieves both ends. This thesis defines the Distributed Pattern Matching (DPM) problem. The DPM problem is to discover a pattern (\ie bit-vector) using any subset of its 1-bits, under the assumption that the patterns are distributed across a large population of networked nodes. Search problem in many distributed systems can be reduced to the DPM problem. This thesis also presents two distinct search mechanisms, named Distributed Pattern Matching System (DPMS) and Plexus, for solving the DPM problem. DPMS is a semi-structured, hierarchical architecture aiming to discover a predefined number of matches by visiting a small number of nodes. Plexus, on the other hand, is a structured search mechanism based on the theory of Error Correcting Code (ECC). The design goal behind Plexus is to discover all the matches by visiting a reasonable number of nodes

SoS: self-organizing substrates

Large-scale networked systems often, both by design or chance exhibit self-organizing properties. Understanding self-organization using tools from cybernetics, particularly modeling them as Markov processes is a first step towards a formal framework which can be used in (decentralized) systems research and design.Interesting aspects to look for include the time evolution of a system and to investigate if and when a system converges to some absorbing states or stabilizes into a dynamic (and stable) equilibrium and how it performs under such an equilibrium state. Such a formal framework brings in objectivity in systems research, helping discern facts from artefacts as well as providing tools for quantitative evaluation of such systems. This thesis introduces such formalism in analyzing and evaluating peer-to-peer (P2P) systems in order to better understand the dynamics of such systems which in turn helps in better designs. In particular this thesis develops and studies the fundamental building blocks for a P2P storage system. In the process the design and evaluation methodology we pursue illustrate the typical methodological approaches in studying and designing self-organizing systems, and how the analysis methodology influences the design of the algorithms themselves to meet system design goals (preferably with quantifiable guarantees). These goals include efficiency, availability and durability, load-balance, high fault-tolerance and self-maintenance even in adversarial conditions like arbitrarily skewed and dynamic load and high membership dynamics (churn), apart of-course the specific functionalities that the system is supposed to provide. The functionalities we study here are some of the fundamental building blocks for various P2P applications and systems including P2P storage systems, and hence we call them substrates or base infrastructure. These elemental functionalities include: (i) Reliable and efficient discovery of resources distributed over the network in a decentralized manner; (ii) Communication among participants in an address independent manner, i.e., even when peers change their physical addresses; (iii) Availability and persistence of stored objects in the network, irrespective of availability or departure of individual participants from the system at any time; and (iv) Freshness of the objects/resources' (up-to-date replicas). Internet-scale distributed index structures (often termed as structured overlays) are used for discovery and access of resources in a decentralized setting. We propose a rapid construction from scratch and maintenance of the P-Grid overlay network in a self-organized manner so as to provide efficient search of both individual keys as well as a whole range of keys, doing so providing good load-balancing characteristics for diverse kind of arbitrarily skewed loads - storage and replication, query forwarding and query answering loads. For fast overlay construction we employ recursive partitioning of the key-space so that the resulting partitions are balanced with respect to storage load and replication. The proper algorithmic parameters for such partitioning is derived from a transient analysis of the partitioning process which has Markov property. Preservation of ordering information in P-Grid such that queries other than exact queries, like range queries can be efficiently and rather trivially handled makes P-Grid suitable for data-oriented applications. Fast overlay construction is analogous to building an index on a new set of keys making P-Grid suitable as the underlying indexing mechanism for peer-to-peer information retrieval applications among other potential applications which may require frequent indexing of new attributes apart regular updates to an existing index. In order to deal with membership dynamics, in particular changing physical address of peers across sessions, the overlay itself is used as a (self-referential) directory service for maintaining the participating peers' physical addresses across sessions. Exploiting this self-referential directory, a family of overlay maintenance scheme has been designed with lower communication overhead than other overlay maintenance strategies. The notion of dynamic equilibrium study for overlays under continuous churn and repairs, modeled as a Markov process, was introduced in order to evaluate and compare the overlay maintenance schemes. While the self-referential directory was originally invented to realize overlay maintenance schemes with lower overheads than existing overlay maintenance schemes, the self-referential directory is generic in nature and can be used for various other purposes, e.g., as a decentralized public key infrastructure. Persistence of peer identity across sessions, in spite of changes in physical address, provides a logical independence of the overlay network from the underlying physical network. This has many other potential usages, for example, efficient maintenance mechanisms for P2P storage systems and P2P trust and reputation management. We specifically look into the dynamics of maintaining redundancy for storage systems and design a novel lazy maintenance strategy. This strategy is algorithmically a simple variant of existing maintenance strategies which adapts to the system dynamics. This randomized lazy maintenance strategy thus explores the cost-performance trade-offs of the storage maintenance operations in a self-organizing manner. We model the storage system (redundancy), under churn and maintenance, as a Markov process. We perform an equilibrium study to show that the system operates in a more stable dynamic equilibrium with our strategy than for the existing maintenance scheme for comparable overheads. Particularly, we show that our maintenance scheme provides substantial performance gains in terms of maintenance overhead and system's resilience in presence of churn and correlated failures. Finally, we propose a gossip mechanism which works with lower communication overhead than existing approaches for communication among a relatively large set of unreliable peers without assuming any specific structure for their mutual connectivity. We use such a communication primitive for propagating replica updates in P2P systems, facilitating management of mutable content in P2P systems. The peer population affected by a gossip can be modeled as a Markov process. Studying the transient spread of gossips help in choosing proper algorithm parameters to reduce communication overhead while guaranteeing coverage of online peers. Each of these substrates in themselves were developed to find practical solutions for real problems. Put together, these can be used in other applications, including a P2P storage system with support for efficient lookup and inserts, membership dynamics, content mutation and updates, persistence and availability. Many of the ideas have already been implemented in real systems and several others are in the way to be integrated into the implementations. There are two principal contributions of this dissertation. It provides design of the P2P systems which are useful for end-users as well as other application developers who can build upon these existing systems. Secondly, it adapts and introduces the methodology of analysis of a system's time-evolution (tools typically used in diverse domains including physics and cybernetics) to study the long run behavior of P2P systems, and uses this methodology to (re-)design appropriate algorithms and evaluate them. We observed that studying P2P systems from the perspective of complex systems reveals their inner dynamics and hence ways to exploit such dynamics for suitable or better algorithms. In other words, the analysis methodology in itself strongly influences and inspires the way we design such systems. We believe that such an approach of orchestrating self-organization in internet-scale systems, where the algorithms and the analysis methodology have strong mutual influence will significantly change the way future such systems are developed and evaluated. We envision that such an approach will particularly serve as an important tool for the nascent but fast moving P2P systems research and development community

Flexible Application-Layer Multicast in Heterogeneous Networks

This work develops a set of peer-to-peer-based protocols and extensions in order to provide Internet-wide group communication. The focus is put to the question how different access technologies can be integrated in order to face the growing traffic load problem. Thereby, protocols are developed that allow autonomous adaptation to the current network situation on the one hand and the integration of WiFi domains where applicable on the other hand

A novel service discovery model for decentralised online social networks.

Online social networks (OSNs) have become the most popular Internet application that attracts billions of users to share information, disseminate opinions and interact with others in the online society. The unprecedented growing popularity of OSNs naturally makes using social network services as a pervasive phenomenon in our daily life. The majority of OSNs service providers adopts a centralised architecture because of its management simplicity and content controllability. However, the centralised architecture for large-scale OSNs applications incurs costly deployment of computing infrastructures and suffers performance bottleneck. Moreover, the centralised architecture has two major shortcomings: the single point failure problem and the lack of privacy, which challenges the uninterrupted service provision and raises serious privacy concerns. This thesis proposes a decentralised approach based on peer-to-peer (P2P) networks as an alternative to the traditional centralised architecture. Firstly, a self-organised architecture with self-sustaining social network adaptation has been designed to support decentralised topology maintenance. This self-organised architecture exhibits small-world characteristics with short average path length and large average clustering coefficient to support efficient information exchange. Based on this self-organised architecture, a novel decentralised service discovery model has been developed to achieve a semantic-aware and interest-aware query routing in the P2P social network. The proposed model encompasses a service matchmaking module to capture the hidden semantic information for query-service matching and a homophily-based query processing module to characterise user’s common social status and interests for personalised query routing. Furthermore, in order to optimise the efficiency of service discovery, a swarm intelligence inspired algorithm has been designed to reduce the query routing overhead. This algorithm employs an adaptive forwarding strategy that can adapt to various social network structures and achieves promising search performance with low redundant query overhead in dynamic environments. Finally, a configurable software simulator is implemented to simulate complex networks and to evaluate the proposed service discovery model. Extensive experiments have been conducted through simulations, and the obtained results have demonstrated the efficiency and effectiveness of the proposed model.University of Derb