Towards a Framework for DHT Distributed Computing

Distributed Hash Tables (DHTs) are protocols and frameworks used by peer-to-peer (P2P) systems. They are used as the organizational backbone for many P2P file-sharing systems due to their scalability, fault-tolerance, and load-balancing properties. These same properties are highly desirable in a distributed computing environment, especially one that wants to use heterogeneous components. We show that DHTs can be used not only as the framework to build a P2P file-sharing service, but as a P2P distributed computing platform. We propose creating a P2P distributed computing framework using distributed hash tables, based on our prototype system ChordReduce. This framework would make it simple and efficient for developers to create their own distributed computing applications. Unlike Hadoop and similar MapReduce frameworks, our framework can be used both in both the context of a datacenter or as part of a P2P computing platform. This opens up new possibilities for building platforms to distributed computing problems. One advantage our system will have is an autonomous load-balancing mechanism. Nodes will be able to independently acquire work from other nodes in the network, rather than sitting idle. More powerful nodes in the network will be able use the mechanism to acquire more work, exploiting the heterogeneity of the network. By utilizing the load-balancing algorithm, a datacenter could easily leverage additional P2P resources at runtime on an as needed basis. Our framework will allow MapReduce-like or distributed machine learning platforms to be easily deployed in a greater variety of contexts

Performance analysis of structured peer-to-peer overlays for mobile networks

Distributed Hash Table (DHT) based Peer-to-Peer (P2P) overlays have been widely researched and deployed in many applications such as file sharing, IP telephony, content distribution and media streaming applications. However, their deployment has largely been restricted to fixed, wired networks. This is due to the fact that supporting P2P overlays on wireless networks such as the public mobile data network is more challenging due to constraints in terms of data transmissions on cellular networks, limited battery power of the handsets and increased levels of node churn. However, the proliferation of smartphones makes the use of P2P applications on mobile handsets very desirable.  In this paper, we have analysed and evaluated the performance and efficiency of five popular DHT based structured P2P overlays (Chord, Pastry, Kademlia, Broose and EpiChord) under conditions as commonly experienced in public mobile data networks. Our results show that the conditions in mobile networks, including a high churn rate and the relatively low bandwidth availability is best matched by Kademlia and EpiChord. These overlays exhibit a high lookup success ratio and low hop count while consuming a moderate amount of bandwidth. These characteristics make these two overlays suitable candidates for use in mobile networks

Content-based addressing in hierarchical distributed hash tables

Peer-to-peer networks have drawn their strength from their ability to operate functionally without the use of a central agent. In recent years the development of the structured peer-to-peer network has further increased the distributed nature of p2p systems. These networks take advantage of an underlying distributed data structure, a common one is the distributed hash table (DHT). These peers use this structure to act as equals in a network, sharing the same responsibilities of maintaining and contributing. But herein lays the problem, not all peers are equal in terms of resources and power. And with no central agent to monitor and balance load , the heterogeneous nature of peers can cause many distribution or bottleneck issues on the network and peer levels. This is due to the way in which addresses are allocated in these DHTs. Often this function is carried out by a consistent hashing function. These functions although powerful in their simplicity and effectiveness are the stem of a crucial flaw. This flaw causes the random nature in which addresses are assigned both when considering peer identification and allocating resource ownership. This work proposes a solution to mitigate the random nature of address assignment in DHTs, leveraging two methodologies called hierarchical DHTs and content based addressing. Combining these methods would enable peers to work in cooperative groups of like interested peers in order to dynamically share the load between group members. Group formation and utilization relies on the actual resources a peer willingly shares and is able to contribute rather than a function of the random hash employed by traditional DHT p2p structures

Whanaungatanga: Sybil-proof routing with social networks

Decentralized systems, such as distributed hash tables, are subject to the Sybil attack, in which an adversary creates many false identities to increase its influence. This paper proposes a routing protocol for a distributed hash table that is strongly resistant to the Sybil attack. This is the first solution to this problem with sublinear run time and space usage. The protocol uses the social connections between users to build routing tables that enable Sybil-resistant distributed hash table lookups. With a social network of N well-connected honest nodes, the protocol can tolerate up to O(N/log N) "attack edges" (social links from honest users to phony identities). This means that an adversary has to fool a large fraction of the honest users before any lookups will fail. The protocol builds routing tables that contain O(N log^(3/2) N) entries per node. Lookups take O(1) time. Simulation results, using social network graphs from LiveJournal, Flickr, and YouTube, confirm the analytical results

A Balanced Trust-Based Method to Counter Sybil and Spartacus Attacks in Chord

A Sybil attack is one of the main challenges to be addressed when securing peer-to-peer networks, especially those based on Distributed Hash Tables (DHTs). Tampering routing tables by means of multiple fake identities can make routing, storing, and retrieving operations significantly more difficult and time-consuming. Countermeasures based on trust and reputation have already proven to be effective in some contexts, but one variant of the Sybil attack, the Spartacus attack, is emerging as a new threat and its effects are even riskier and more difficult to stymie. In this paper, we first improve a well-known and deployed DHT (Chord) through a solution mixing trust with standard operations, for facing a Sybil attack affecting either routing or storage and retrieval operations. This is done by maintaining the least possible overhead for peers. Moreover, we extend the solution we propose in order for it to be resilient also against a Spartacus attack, both for an iterative and for a recursive lookup procedure. Finally, we validate our findings by showing that the proposed techniques outperform other trust-based solutions already known in the literature as well

Building Robust Distributed Infrastructure Networks

Many competing designs for Distributed Hash Tables exist exploring multiple models of addressing, routing and network maintenance. Designing a general theoretical model and implementation of a Distributed Hash Table allows exploration of the possible properties of Distributed Hash Tables. We will propose a generalized model of DHT behavior, centered on utilizing Delaunay triangulation in a given metric space to maintain the networks topology. We will show that utilizing this model we can produce network topologies that approximate existing DHT methods and provide a starting point for further exploration. We will use our generalized model of DHT construction to design and implement more efficient Distributed Hash Table protocols, and discuss the qualities of potential successors to existing DHT technologies

An interoperable and secure architecture for internet-scale decentralized personal communication

Interpersonal network communications, including Voice over IP (VoIP) and Instant Messaging (IM), are increasingly popular communications tools. However, systems to date have generally adopted a client-server model, requiring complex centralized infrastructure, or have not adhered to any VoIP or IM standard. Many deployment scenarios either require no central equipment, or due to unique properties of the deployment, are limited or rendered unattractive by central servers. to address these scenarios, we present a solution based on the Session Initiation Protocol (SIP) standard, utilizing a decentralized Peer-to-Peer (P2P) mechanism to distribute data. Our new approach, P2PSIP, enables users to communicate with minimal or no centralized servers, while providing secure, real-time, authenticated communications comparable in security and performance to centralized solutions.;We present two complete protocol descriptions and system designs. The first, the SOSIMPLE/dSIP protocol, is a P2P-over-SIP solution, utilizing SIP both for the transport of P2P messages and personal communications, yielding an interoperable, single-stack solution for P2P communications. The RELOAD protocol is a binary P2P protocol, designed for use in a SIP-using-P2P architecture where an existing SIP application is modified to use an additional, binary RELOAD stack to distribute user information without need for a central server.;To meet the unique security needs of a fully decentralized communications system, we propose an enrollment-time certificate authority model that provides asserted identity and strong P2P and user-level security. In this model, a centralized server is contacted only at enrollment time. No run-time connections to the servers are required.;Additionally, we show that traditional P2P message routing mechanisms are inappropriate for P2PSIP. The existing mechanisms are generally optimized for file sharing and neglect critical practical elements of the open Internet --- namely link-level security and asymmetric connectivity caused by Network Address Translators (NATs). In response to these shortcomings, we introduce a new message routing paradigm, Adaptive Routing (AR), and using both analytical models and simulation show that AR significantly improves message routing performance for P2PSIP systems.;Our work has led to the creation of a new research topic within the P2P and interpersonal communications communities, P2PSIP. Our seminal publications have provided the impetus for subsequent P2PSIP publications, for the listing of P2PSIP as a topic in conference calls for papers, and for the formation of a new working group in the Internet Engineering Task Force (IETF), directed to develop an open Internet standard for P2PSIP

A Content-Addressable Network for Similarity Search in Metric Spaces

Because of the ongoing digital data explosion, more advanced search paradigms than the traditional exact match are needed for contentbased retrieval in huge and ever growing collections of data produced in application areas such as multimedia, molecular biology, marketing, computer-aided design and purchasing assistance. As the variety of data types is fast going towards creating a database utilized by people, the computer systems must be able to model human fundamental reasoning paradigms, which are naturally based on similarity. The ability to perceive similarities is crucial for recognition, classification, and learning, and it plays an important role in scientific discovery and creativity. Recently, the mathematical notion of metric space has become a useful abstraction of similarity and many similarity search indexes have been developed. In this thesis, we accept the metric space similarity paradigm and concentrate on the scalability issues. By exploiting computer networks and applying the Peer-to-Peer communication paradigms, we build a structured network of computers able to process similarity queries in parallel. Since no centralized entities are used, such architectures are fully scalable. Specifically, we propose a Peer-to-Peer system for similarity search in metric spaces called Metric Content-Addressable Network (MCAN) which is an extension of the well known Content-Addressable Network (CAN) used for hash lookup. A prototype implementation of MCAN was tested on real-life datasets of image features, protein symbols, and text — observed results are reported. We also compared the performance of MCAN with three other, recently proposed, distributed data structures for similarity search in metric spaces

Enabling technologies for decentralized interpersonal communication

In the recent years the Internet users have witnessed the emergence of Peer-to-Peer (P2P) technologies and applications. One class of P2P applications is comprised of applications that are targeted for interpersonal communication. The communication applications that utilize P2P technologies are referred to as decentralized interpersonal communication applications. Such applications are decentralized in a sense that they do not require assistance from centralized servers for setting up multimedia sessions between users. The invention of Distributed Hash Table (DHT) algorithms has been an important, but not an inclusive enabler for decentralized interpersonal communication. Even though the DHTs provide a basic foundation for decentralization, there are still a number of challenges without viable technological solutions. The main contribution of this thesis is to propose technological solutions to a subset of the existing challenges. In addition, this thesis also presents the preliminary work for the technological solutions. There are two parts in the preliminary work. In the first part, a set of DHT algorithms are evaluated from the viewpoint of decentralized interpersonal communication, and the second part gives a coherent presentation of the challenges that a decentralized interpersonal communication application is going to encounter in mobile networks. The technological solution proposals contain two architectures and two algorithms. The first architecture enables an interconnection between a decentralized and a centralized communication network, and the second architecture enables the decentralization of a set of legacy applications. The first algorithm is a load balancing algorithm that enables good scalability, and the second algorithm is a search algorithm that enables arbitrary searches. The algorithms can be used, for example, in DHT-based networks. Even though this thesis has focused on the decentralized interpersonal communication, some of the proposed technological solutions also have general applicability outside the scope of decentralized interpersonal communication