Gossip-based self-managing services for large scale dynamic networks

Modern IP networks are dynamic, large-scale and heterogeneous. This implies that they are more unpredictable and difficult to maintain and build upon. Implementation and management of decentralized applications that exploit these networks can be enabled only through a set of special middleware services that shield the application from the scale, dynamism and heterogeneity of the environment. Among others, these services have to provide communication services (routing, multicasting, etc.) and global information like network size, load distribution, etc. The goal is not to provide abstractions that hide the distributedness of the system, but rather, to hide the unpleasant features of the system, such as dynamism, scale and heterogeneity. Most importantly, these services have to be self-managing: they have to be able to maintain certain properties in the face of extreme dynamism of the network. In this manner, such services can serve as the lowest layer that makes possible building more complex applications, or simply as a plugin to enhance existing systems, for example, GRID environments. Apart from self-management, we require that the services be simple and lightweight, to allow easy implementation and incur low cost. Our approach to achieving these goals is based on the gossip communication model. Gossip protocols are simple, robust and scalable, besides, they can be applied to implement not only information dissemination, but several other functions, as we will show. So far, we have designed gossip-based protocols for maintaining random overlays, which define group membership. Based on this random overlay, we have designed gossip-based protocols to calculate aggregate values such as maxima, average, sum, variance, etc. We have also developed protocols to build several structured overlays in this framework, including superpeer, torus, ring, binary tree, etc. These protocols build on the random overlay and also on aggregate values. The gossip-based model is well suited to dynamic and large networks. Our protocols are extremely simple to implement while being robust and adaptive without adding any extra components or control loops. Our approach also support composition at a local level. At each node in the network, the same services are available: for example, data aggregation uses the random overlay (peer sampling service) and superpeer topology construction applies aggregate values, such as maximal and average capacity. In fact, protocols that implement the different services are heavily interconnected and form a modular system within this lighweight self-managing service layer. While this presentation focuses on the self-managing systems services, it is clear that other application-level services can also be built at higher layers. These services can be proactive, like load balancing, that can make use of the target (average) load and overlays for optimization of load transfer, or reactive, like broadcasting or search, that can be performed on top of an appropriate overlay network (eg spanning tree or superpeer network), maintained by the lighweight self-managing systems services

Towards Data Mining in Large and Fully Distributed Peer-To-Peer Overlay Networks

The Internet, which is becoming a more and more dynamic, extremely heterogeneous network has recently became a platform for huge fully distributed peer-to-peer overlay networks containing millions of nodes typically for the purpose of information dissemination and file sharing. This paper targets the problem of analyzing data which are scattered over a such huge and dynamic set of nodes, where each node is storing possibly very little data but where the total amount of data is immense due to the large number of nodes. We present distributed algorithms for effectively calculating basic statistics of data using the recently introduced newscast model of computation and we demonstrate how to implement basic data mining algorithms based on these techniques. We will argue that the suggested techniques are efficient, robust and scalable and that they preserve the privacy of data

Decentralized Ranking in Large-Scale Overlay Networks

Modern distributed systems are often characterized by very large scale, poor reliability, and extreme dynamism of the participating nodes, with a continuous flow of nodes joining and leaving the system. In order to develop robust applications in such environments, middleware services aimed at dealing with the inherent unpredictability of the underlying networks are required. One such service is aggregation. In the aggregation problem, each node is assumed to have attributes. The task is to extract global information about these attributes and make it available to the nodes. Examples include the total free storage, the average load, or the size of the network. Efficient protocols for computing several aggregates such as average, count, and variance have already been proposed. In this paper, we consider calculating the rank of nodes, where the set of nodes has to be sorted according to a numeric attribute and each node must be informed about its own rank in the global sorting. This information has a number of applications, such as slicing. It can also be applied to calculate the median or any other percentile. We propose T-Rank, a robust and completely decentralized algorithm for solving the ranking problem with minimal assumptions. Due to the characteristics of the targeted environment, we aim for a probabilistic approach and accept minor errors in the output. We present extensive empirical results that suggest near logarithmic time complexity, scalability and robustness in different failure scenarios

A Wave Analysis of the Subset Sum Problem

This paper introduces the wave model, a novel approach on analyzing the behavior of GAs. Our aim is to give techniques that have practical relevance and provide tools for improving the performance of the GA or for discovering simple and effective heuristics on certain problem classes. The wave analysis is the process of building wave models of problem instances of a problem class and extracting common features that characterize the problem class in question. A wave model is made of paths which are composed of subsets of the search space (features) that are relevant from the viewpoint of the search. The GA is described as a basicly {\em sequential} process; a wave motion along the paths that form the wave model. The method is demonstrated via an analysis of the NP-complete subset sum problem. Based on the analysis, problem specific GA modifications and a new heuristic will be suggested that outperform the original G

Reflections on Niyogi's book "The Informational Complexity Of Learning"

Niyogi's book entitled "The Informational Complexity of Learning" [6] addresses the problem of learning from examples. He considers two distant fields: artificial neural networks and natural language. In both areas he gives a theoretical analysis of informational complexity, i.e. the effects of the size of the leaning set and the number of model parameters on the accuracy of learning depending on the target function class. After outlining the main ideas, this work discusses the usability of such results in practice and the relevance of the book in linguistic research, and also raises a philosophical question about the possibility of error prediction

The adaptationist stance and evolutionary computation

In this paper the connections between the evolutionary paradigm called adaptationism and the field of evolutionary computation (EC) will be outlined. After giving an introduction to adaptationism we will try to show that the so called adaptational stance can be applied in EC as well as in biology and this application may have significant benefits. It will also be shown that this approach has serious, inherent limitations in both cases especially in the case of EC, because we lack the language which could be used to form the theories, but these representational limitations can be handled by devoting efforts to construct this language

Maintaining Connectivity in a Scalable and Robust Distributed Environment

This paper describes a novel peer-to-peer (P2P) environment for running distributed Java applications on the Internet. The possible application areas include simple load balancing, parallel evolutionary computation, agent-based simulation and artificial life. Our environment is based on cutting-edge P2P technology. We introduce and analyze the concept of long term memory which provides protection against partitioning of the network. We demonstrate the potentials of our approach by analyzing a simple distributed application. We present theoretical and empirical evidence that our approach is scalable, effective and robust