Comprehending Kademlia Routing - A Theoretical Framework for the Hop Count Distribution

The family of Kademlia-type systems represents the most efficient and most widely deployed class of internet-scale distributed systems. Its success has caused plenty of large scale measurements and simulation studies, and several improvements have been introduced. Its character of parallel and non-deterministic lookups, however, so far has prevented any concise formal analysis. This paper introduces the first comprehensive formal model of the routing of the entire family of systems that is validated against previous measurements. It sheds light on the overall hop distribution and lookup delays of the different variations of the original protocol. It additionally shows that several of the recent improvements to the protocol in fact have been counter-productive and identifies preferable designs with regard to routing overhead and resilience.Comment: 12 pages, 6 figure

A Lightweight Approach for Improving the Lookup Performance in Kademlia-type Systems

Discovery of nodes and content in large-scale distributed systems is generally based on Kademlia, today. Understanding Kademlia-type systems to improve their performance is essential for maintaining a high service quality for an increased number of participants, particularly when those systems are adopted by latency-sensitive applications. This paper contributes to the understanding of Kademlia by studying the impact of \emph{diversifying} neighbours' identifiers within each routing table bucket on the lookup performance. We propose a new, yet backward-compatible, neighbour selection scheme that attempts to maximize the aforementioned diversity. The scheme does not cause additional overhead except negligible computations for comparing the diversity of identifiers. We present a theoretical model for the actual impact of the new scheme on the lookup's hop count and validate it against simulations of three exemplary Kademlia-type systems. We also measure the performance gain enabled by a partial deployment for the scheme in the real KAD system. The results confirm the superiority of the systems that incorporate our scheme.Comment: 13 pages, 8 figures, conference version 'Diversity Entails Improvement: A new Neighbour Selection Scheme for Kademlia-type Systems' at IEEE P2P 201

Towards a common architecture to interconnect heterogeneous overlay networks

ICPADS Workshop sessionInternational audienceThis paper presents a novel overlay architecture to allow the design and development of distributed applications based on multiple interconnected overlay networks. Message routing between overlays is achieved via co-located nodes, i.e. nodes that are part of multiple overlay networks at the same time. Co-located nodes, playing the role of distributed gateways, allow a message to reach a wider set of peers while overlay maintenance remains localized to individual overlays of smaller size. To increase robustness, gateway nodes route messages in an unstructured fashion, and can discover each other by analyzing the overlay traffic. The approach is able to work in both "collaborative" scenarios, where overlay protocol messages can be modified to include additional inter-routing information, or non-collaborative ones. This allows for the interaction with existing overlay protocols already deployed

Sub-Second Lookups on a Large-Scale Kademlia-Based Overlay

Previous studies of large-scale (multimillion node) Kademlia-based DHTs have shown poor performance, measured in seconds, in contrast to the far more optimistic results from theoretical analysis, simulations and testbeds. In this paper, we unexpectedly find that in the Mainline BitTorrent DHT (MDHT), probably the largest DHT overlay on the Internet, many lookups already yield results in less than a second, albeit not consistently. With the backwards-compatible modifications we present, we show that not only can we reduce median latencies to between 100 and 200 ms, but also consistently achieve sub-second lookups. These results suggest that it is possible to deploy latency-sensitive applications on top of large-scale DHT overlays on the Internet, contrary to what some might have concluded based on previous results reported in the literature.QC 2011082

Confidential Data-Outsourcing and Self-Optimizing P2P-Networks: Coping with the Challenges of Multi-Party Systems

This work addresses the inherent lack of control and trust in Multi-Party Systems at the examples of the Database-as-a-Service (DaaS) scenario and public Distributed Hash Tables (DHTs). In the DaaS field, it is shown how confidential information in a database can be protected while still allowing the external storage provider to process incoming queries. For public DHTs, it is shown how these highly dynamic systems can be managed by facilitating monitoring, simulation, and self-adaptation