Gestion du déploiement de composants sur réseau P2P

The deployment of component-based applications relies on a centralized directory to store the components. This paper describes an approach to distribute software components to be deployed on a set of peers of a peer to peer network in order to exploit some associated characteristics (load balancing, fault-tolerance, self-organisation). The proposed architecture is situated in the context of OSGI application deployment management. The software components (bundles) are distributed among a set of nodes participating in the execution of services. When a node wants to install a component which is not deployed locally, the component is looked for and installed using a p2p network. –––––Le déploiement d'applications à composants repose sur une approche d'annuaire centralisé de stockage des composants. Cet article décrit une approche pour distribuer les composants logiciels à déployer sur un ensemble de noeuds d'un réseau pair-à-pair afin de pouvoir exploiter certaines caractéristiques associées (équilibrage de charge, tolérance de panne, auto-organisation). L'architecture proposée entre dans le cadre de la gestion du déploiement d'applications sur le modèle OSGi. Les composants logiciels (ou bundles) sont répartis à travers un ensemble de noeuds participant à l'exécution de services. Lorsqu'un noeud veut installer un composant et si celui-ci n'est pas encore déployé localement, il est recherché et installé en utilisant un réseau p2

Storing and managing data in a distributed hash table

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2008.Includes bibliographical references (p. 83-90).Distributed hash tables (DHTs) have been proposed as a generic, robust storage infrastructure for simplifying the construction of large-scale, wide-area applications. For example, UsenetDHT is a new design for Usenet News developed in this thesis that uses a DHT to cooperatively deliver Usenet articles: the DHT allows a set of N hosts to share storage of Usenet articles, reducing their combined storage requirements by a factor of O(N). Usenet generates a continuous stream of writes that exceeds 1 Tbyte/day in volume, comprising over ten million writes. Supporting this and the associated read workload requires a DHT engineered for durability and efficiency. Recovering from network and machine failures efficiently poses a challenge for DHT replication maintenance algorithms that provide durability. To avoid losing the last replica, replica maintenance must create additional replicas when failures are detected. However, creating replicas after every failure stresses network and storage resources unnecessarily. Tracking the location of every replica of every object would allow a replica maintenance algorithm to create replicas only when necessary, but when storing terabytes of data, such tracking is difficult to perform accurately and efficiently. This thesis describes a new algorithm, Passing Tone, that maintains durability efficiently, in a completely decentralized manner, despite transient and permanent failures. Passing Tone nodes make replication decisions with just basic DHT routing state, without maintaining state about the number or location of extant replicas and without responding to every transient failure with a new replica. Passing Tone is implemented in a revised version of DHash, optimized for both disk and network performance.(cont.) A sample 12 node deployment of Passing Tone and UsenetDHT supports a partial Usenet feed of 2.5 Mbyte/s (processing over 80 Tbyte of data per year), while providing 30 Mbyte/s of read throughput, limited currently by disk seeks. This deployment is the first public DHT to store terabytes of data. These results indicate that DHT-based designs can successfully simplify the construction of large-scale, wide-area systems.by Emil Sit.Ph.D

DHash table

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, February 2006.Includes bibliographical references (p. 123-132) and index.DHash is a new system that harnesses the storage and network resources of computers distributed across the Internet by providing a wide-area storage service, DHash. DHash frees applications from re-implementing mechanisms common to any system that stores data on a collection of machines: it maintains a mapping of objects to servers, replicates data for durability, and balances load across participating servers. Applications access data stored in DHash through a familiar hash-table interface: put stores data in the system under a key; get retrieves the data. DHash has proven useful to a number of application builders and has been used to build a content-distribution system [31], a Usenet replacement [115], and new Internet naming architectures [130, 129]. These applications demand low-latency, high-throughput access to durable data. Meeting this demand is challenging in the wide-area environment. The geographic distribution of nodes means that latencies between nodes are likely to be high: to provide a low-latency get operation the system must locate a nearby copy of the data without traversing high-latency links.(cont.) Also, wide-area network links are likely to be less reliable and have lower capacities than local-area network links: to provide durability efficiently the system must minimize the number of copies of data items it sends over these limited capacity links in response to node failure. This thesis describes the design and implementation of the DHash distributed hash table and presents algorithms and techniques that address these challenges. DHash provides low-latency operations by using a synthetic network coordinate system (Vivaldi) to find nearby copies of data without sending messages over high-latency links. A network transport (STP), designed for applications that contact a large number of nodes, lets DHash provide high throughput by striping a download across many servers without causing high packet loss or exhausting local resources. Sostenuto, a data maintenance algorithm, lets DHash maintain data durability while minimizing the number of copies of data that the system sends over limited-capacity links.by Frank Dabek.Ph.D

Co-operação de Tabelas de Hash Distribuídas em clusters heterogéneos

Tese de Doutoramento em Informática - Área de Conhecimento de Engenharia de ComputadoresAs Estruturas de Dados Distribuídas (DDSs) constituem uma abordagem ao Armazenamento Distribuído adequada a aplicações com requisitos de elevada capacidade de armazenamento, escalabilidade e disponibilidade. Ao mesmo tempo, a disponibilização de interfaces simples e familiares permite encurtar o ciclo de desenvolvimento de aplicações. Num ambiente de cluster, caracterizado pela coexistência de grande variedade de aplicações e utilizadores com requisitos dinâmicos e de gerações de equipamento com desempenhos diferenciados (tanto ao nível do poder de calculo, como das comunicações e do armazenamento), a necessidade de operação conjunta e eficiente de múltiplas DDSs pelas aplicações constituiu um desafio científico e tecnológico que esta tese se propôs superar. Esta tese apresenta as propostas e os resultados alcançados durante o estudo e desenvolvimento dos modelos e da plataforma de suporte à arquitectura Domus para a co-operação de múltiplas Tabelas de Hash Distribuídas (DHTs) em clusters partilhados e heterogéneos. A plataforma contempla várias classes de atributos e operações sobre DHTs permitindo, entre outras possibilidades, a definição administrativa de limite à expansão/contracção, o recurso a diferentes tecnologias de armazenamento e a suspensão/retoma dinâmica da operação. Numa outra vertente, para responder aos requisitos impostos pela execução simultânea de múltiplas e diferenciadas aplicações, foram formulados e desenvolvidos mecanismos de balanceamento dinâmico de carga que visam a optimização e rentabilização dos recursos computacionais/comunicacionais e de armazenamento disponíveis no cluster. Na base da abordagem estão modelos que dão uma resposta qualificada aos problemas do particionamento de uma DHT e da localização das suas entradas. Em relação ao particionamento, são propostos modelos para definição do número de entradas de cada nó de uma DHT, suportando variação dinâmica do número total de nós; os modelos asseguram uma distribuição óptima do número de entradas, nos quatro cenários que resultam da combinação de Hashing Estático ou Dinâmico, com Distribuição Homogénea ou Heterogénea; com Hashing Dinâmico, a qualidade da distribuição é parametrizável e, com Distribuição Homogénea, o esforço de re-particionamento ´e de ordem O(1) face ao total de nós da DHT. No que toca à localização, definiram-se algoritmos de encaminhamento acelerado para localização distribuída em topologias DeBruijn e Chord, que respondem `a descontinuidade das partições das DHTs geradas pelos modelos de particionamento. Para o efeito, explorou-se a possível coexistência em cada nó de uma DHT de múltiplas tabelas de encaminhamento, para tentar aproximar do custo, de referência, sobre grafos esparsos em que os vértices são nós da DHT, ao custo da localização sobre grafos completos, em que os vértices são entradas da DHT, tendo-se obtido valores de 70% a 90% do custo de referência.Distributed Data Structures (DDSs) as an approach to Distributed Storage are adequate to applications that require high storage capacity, scalability and availability. At the same time, DDSs present simple and familiar interfaces that allow shorter development cycles. In a cluster environment, shared by multiple users and applications with dynamic requisites, and built on different hardware generations (with different computing, communication and storage power), the need to simultaneously and efficiently operate several DDSs by user applications presents the scientific and technological challenge embraced by this thesis. This thesis presents the proposals and the results achieved during the study of the models and development of the platform that supports the Domus architecture for the co-operation of multiple Distributed Hash Tables (DHTs) in shared and heterogeneous clusters. The platform supports several classes of attributes and operations on DHTs allowing, among other possibilities, the administrative definition of per DHT node boundaries and storage technologies, or the deactivation/reactivation of DHTs. In addition, Domus DHTs are managed by dynamic load balancing mechanisms which ensure certain service levels (like storage space or access performance) and optimize the utilization of cluster resources. The architecture builds on specific models for the partitioning and lookup of a DHT address space. The partitioning models ensure optimal distribution of the number of DHT buckets per node, for a dynamic number of DHT nodes, and support the different scenarios that arise from the combination of Static/Dynamic Hashing, with Homogeneous/Heterogeneous Distributions. The quality of the distribution is tuneable under Dynamic Hashing, and repartitioning involves O(1) nodes of the DHT under Homogeneous Distributions. With regard to the lookup problem, a set of accelerated routing algorithms were designed for distributed lookup, both for DeBruijn and Chord topologies, in order to deal with the discontinuous DHT partitions generated by our partitioning models. The algorithms explore the availability of multiple routing tables per DHT node with the aim of reducing the lookup cost with full graphs, in which the vertexes are all the DHT buckets, towards the reference lookup cost with sparse graphs, in which the vertexes are all the DHT nodes. The best lookup algorithms ensure a lookup cost of 70% to 90% of the reference cost

Co-operação de tabelas de Hash distribuídas em clusters heterogéneos

As Estruturas de Dados Distribuídas (DDSs) constituem uma abordagem ao Armazenamento Distribuído adequada a aplicações com requisitos de elevada capacidade de armazenamento, escalabilidade e disponibilidade. Ao mesmo tempo, apresentando-se com interfaces simples e familiares, permitem encurtar o ciclo de desenvolvimento de aplicações. Num ambiente de cluster, caracterizado pela co-existência de grande variedade de aplicações e utilizadores com requisitos dinâmicos, e de gerações de equipamento com desempenhos diferenciados (tanto ao nível do poder de cálculo, como das comunicações e do armazenamento), a necessidade de operação conjunta e eficiente de múltiplas DDSs pelas aplicações constituiu um desafio científico e tecnológico que esta tese se propôs superar. Esta tese apresenta as propostas e os resultados alcançados durante o estudo e desenvolvimento dos modelos e da plataforma de suporte à arquitectura Domus para a co-operação de múltiplas Tabelas de Hash Distribuídas (DHTs) em clusters partilhados e heterogéneos. A plataforma contempla várias classes de atributos e operações sobre DHTs permitindo, entre outras possibilidades, a definição administrativa de limites à expansão/contracção, o recurso a diferentes tecnologias de armazenamento e a suspensão/retoma administrativa da operação. Numa outra vertente, para responder aos requisitos impostos pela execução simultânea de múltiplas e diferenciadas aplicações, foram formulados e desenvolvidos mecanismos de balanceamento dinâmico de carga que visam a optimização e rentabilização dos recursos computacionais, comunicacionais e de armazenamento disponíveis no cluster. Na base da abordagem estão modelos que dão uma resposta qualificada aos problemas do particionamento e da localização de entradas numa DHT. Em relação ao particionamento, são propostos modelos de distribuição para definição do número de entradas de cada nó de uma DHT, suportando variação dinâmica do número total de nós; os modelos asseguram uma distribuição óptima do número de entradas, nos quatro cenários que resultam da combinação de Hashing Estático ou Dinâmico, com Distribuição Homogénea ou Heterogénea; com Hashing Dinâmico, a qualidade da distribuição é parametrizável e, com Distribuição Homogénea, o esforço de re-distribuição é de ordem O(1) face ao total de nós da DHT. No que toca à localização, definiram-se algoritmos de encaminhamento acelerado para localização distribuída em topologias DeBruijn e Chord, que respondem à descontinuidade das partições das DHTs geradas pelos modelos de distribuição. Para o efeito, explorou-se a possível coexistência de múltiplas tabelas de encaminhamento em cada nó de uma DHT, para tentar aproximar o maior custo da localização sobre grafos completos (em que os vértices são entradas da DHT), ao custo de referência sobre grafos esparsos (em que os vértices são nós da DHT), tendo-se obtido valores de 70% a 90% do custo de referência.Distributed Data Structures (DDSs) as an approach to Distributed Storage are adequate to applications that require high storage capacity, scalability and availability. At the same time, DDSs present simple and familiar interfaces that allow shorter development cycles. In a cluster environment, shared by multiple users and applications with dynamic requisites, and built on different hardware generations (with different computing, communication and storage power), the need to simultaneously and efficiently operate several DDSs by user applications presents the scientific and technological challenge embraced by this thesis. This thesis presents the proposals and the results achieved during the study of the models and development of the platform that supports the Domus architecture for the co-operation of multiple Distributed Hash Tables (DHTs) in shared and heterogeneous clusters. The platform supports several classes of attributes and operations on DHTs allowing, among other possibilities, the administrative definition of per DHT node boundaries and storage technologies, or the deactivation/reactivation of DHTs. In addition, Domus DHTs are managed by dynamic load balancing mechanisms which ensure certain service levels (like storage space or access performance) and optimize the utilization of cluster resources. The architecture builds on specific models for the partitioning and lookup of a DHT address space. The partitioning models ensure optimal distribution of the number of DHT buckets per node, for a dynamic number of DHT nodes, and support the different scenarios that arise from the combination of Static/Dynamic Hashing, with Homogeneous/Heterogeneous Distributions. The quality of the distribution is tuneable under Dynamic Hashing, and repartitioning involves O(1) nodes of the DHT under Homogeneous Distributions.PRODEP III/5.3/N/199.006/0

Reorganization in network regions for optimality and fairness

Thesis (S.M.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2004.Includes bibliographical references (p. 92-95).(cont.) down implicit assumptions of altruism while showing the resulting negative impact on utility. From a selfish equilibrium, with much lower global utility, we show the ability of our algorithm to reorganize and restore the utility of individual nodes, and the system as a whole, to similar levels as realized in the SuperPeer network. Simulation of our algorithm shows that it reaches the predicted optimal utility while providing fairness not realized in other systems. Further analysis includes an epsilon equilibrium model where we attempt to more accurately represent the actual reward function of nodes. We find that by employing such a model, over 60% of the nodes are connected. In addition, this model converges to a utility 34% greater than achieved in the SuperPeer network while making no assumptions on the benevolence of nodes or centralized organization.This thesis proposes a reorganization algorithm, based on the region abstraction, to exploit the natural structure in overlays that stems from common interests. Nodes selfishly adapt their connectivity within the overlay in a distributed fashion such that the topology evolves to clusters of users with shared interests. Our architecture leverages the inherent heterogeneity of users and places within the system their incentives and ability to affect the network. As such, it is not dependent on the altruism of any other nodes in the system. Of particular interest is the optimality and fairness of our design. We rigorously define ideal and fair networks and develop a continuum of optimality measures by which to evaluate our algorithm. Further, to evaluate our algorithm within a realistic context, validate assumptions and make design decisions, we capture data from a portion of a live file-sharing network. More importantly, we discover, name, quantify and solve several previously unrecognized subtle problems in a content-based self-organizing network as a direct result of simulations using the trace data. We motivate our design by examining the dependence of existing systems on benevolent Super-Peers. Through simulation we find that the current architecture is highly dependent on the filtering capability and the willingness of the SuperPeer network to absorb the majority of the query burden. The remainder of the thesis is devoted to a world in which SuperPeers no longer exist or are untenable. In our evaluation, we introduce four reasons for utility suboptimal self-reorganizing networks: anarchy (selfish behavior), indifference, myopia and ordering. We simulate the level of utility and happiness achieved in existing architectures. Then we systematically tearby Robert E. Beverly, IV.S.M

Design and Evaluation of Distributed Algorithms for Placement of Network Services

Network services play an important role in the Internet today. They serve as data caches for websites, servers for multiplayer games and relay nodes for Voice over IP: VoIP) conversations. While much research has focused on the design of such services, little attention has been focused on their actual placement. This placement can impact the quality of the service, especially if low latency is a requirement. These services can be located on nodes in the network itself, making these nodes supernodes. Typically supernodes are selected in either a proprietary or ad hoc fashion, where a study of this placement is either unavailable or unnecessary. Previous research dealt with the only pieces of the problem, such as finding the location of caches for a static topology, or selecting better routes for relays in VoIP. However, a comprehensive solution is needed for dynamic applications such as multiplayer games or P2P VoIP services. These applications adapt quickly and need solutions based on the immediate demands of the network. In this thesis we develop distributed algorithms to assign nodes the role of a supernode. This research first builds off of prior work by modifying an existing assignment algorithm and implementing it in a distributed system called Supernode Placement in Overlay Topologies: SPOT). New algorithms are developed to assign nodes the supernode role. These algorithms are then evaluated in SPOT to demonstrate improved SN assignment and scalability. Through a series of simulation, emulation, and experimentation insight is gained into the critical issues associated with allocating resources to perform the role of supernodes. Our contributions include distributed algorithms to assign nodes as supernodes, an open source fully functional distributed supernode allocation system, an evaluation of the system in diverse networking environments, and a simulator called SPOTsim which demonstrates the scalability of the system to thousands of nodes. An example of an application deploying such a system is also presented along with the empirical results

Spurring Adoption of DHTs with OpenHash, a Public DHT Service

this paper is an examination of what one can do to extend the functionality of a DHT service so that such application-specific requirements can be reasonably accommodated. That is, we seek to share a common DHT routing platform while allowing application-specific functionality to reside only on certain nodes. To meet this hybrid challenge of shared routing but application-specific processing we propose ReDiR, a distributed rendezvous scheme that removes the need in today's DHT systems to place applicationspecific code together with DHT code. The ReDiR algorithm itself requires only a put()/get() interface from a DHT, but allows a surprisingly wide range of applications to work over a DHT servic