Small-world networks, distributed hash tables and the e-resource discovery problem

Resource discovery is one of the most important underpinning problems behind producing a scalable, robust and efficient global infrastructure for e-Science. A number of approaches to the resource discovery and management problem have been made in various computational grid environments and prototypes over the last decade. Computational resources and services in modern grid and cloud environments can be modelled as an overlay network superposed on the physical network structure of the Internet and World Wide Web. We discuss some of the main approaches to resource discovery in the context of the general properties of such an overlay network. We present some performance data and predicted properties based on algorithmic approaches such as distributed hash table resource discovery and management. We describe a prototype system and use its model to explore some of the known key graph aspects of the global resource overlay network - including small-world and scale-free properties

Efficient Probabilistic Subsumption Checking for Content-Based Publish/Subscribe Systems

Abstract. Efficient subsumption checking, deciding whether a subscription or publication is covered by a set of previously defined subscriptions, is of paramount importance for publish/subscribe systems. It provides the core system functionality—matching of publications to subscriber needs expressed as subscriptions—and additionally, reduces the overall system load and generated traffic since the covered subscriptions are not propagated in distributed environments. As the subsumption problem was shown previously to be co-NP complete and existing solutions typically apply pairwise comparisons to detect the subsumption relationship, we propose a ‘Monte Carlo type ’ probabilistic algorithm for the general subsumption problem. It determines whether a publication/subscription is covered by a disjunction of subscriptions in O(k md), wherek is the number of subscriptions, m is the number of distinct attributes in subscriptions, and d is the number of tests performed to answer a subsumption question. The probability of error is problem-specific and typically very small, and sets an upper bound on d. Our experimental results show significant gains in term of subscription set reduction which has favorable impact on the overall system performance as it reduces the total computational costs and networking traffic. Furthermore, the expected theoretical bounds underestimate algorithm performance because it performs much better in practice due to introduced optimizations, and is adequate for fast forwarding of subscriptions in case of high subscription rate.

Arigatoni: A Simple Programmable Overlay Network

International audienceWe design a lightweight Overlay Network, called Arigatoni, that is suitable to deploy the Global Computing Paradigm over the Internet. Communications over the behavioral units of the model are performed by a simple communication protocol. Basic Global Computers can communicate by first registering to a brokering service and then by mutually asking and offering services, in a way that is reminiscent to Rapoport's "tit-for-tat" strategy of cooperation based on reciprocity. In the model, resources are encapsulated in the administrative domain in which they reside, and requests for resources located in another administrative domain traverse a broker-2-broker negotiation using classical PKI mechanisms. The model is suitable to fit with various global scenarios from classical P2P applications, like file sharing, or band-sharing, to more sophisticated Grid applications, like remote and distributed big (and small) computations, to possible, futuristic real migrating computations. Indeed, our model fits some of the objectives suggested by the CoreGrid Network of Excellence, as described in Schwiegelshohn et al. (Schwiegelshohn et al., 2005)

Fast Probabilistic Subsumption Checking for Publish/Subscribe Systems

Efficient subsumption checking, deciding whether a subscription or publication is subsumed (covered) by a set of previously defined subscriptions, is of paramount importance for publish/subscribe systems. It provides the core system functionality, and additionally, reduces the overall system load and generated traffic in distributed environments. As the deterministic solution was shown previously to be co-NP complete and existing solutions typically employ costly pairwise comparisons to detect the subsumption relationship, we propose a probabilistic algorithm for the general subsumption problem. It efficiently determines whether a publication/subscription is covered by a disjunction of subscriptions in $O(k~m~d)$, where $k$ is the number of subscriptions, $m$ is the number of distinct attributes in subscriptions, and $d$ is the number of tests performed to answer a subsumption question. The probability of error is problem specific and typically very small, and determines an upper bound on $d$ in polynomial time prior to the algorithm execution. Our experimental results demonstrate the algorithm performs even better in practice due to introduced optimizations, and is adequate for fast forwarding of publications/subscriptions, especially in resource scarce environments, e.g. sensor networks

A Middleware framework for self-adaptive large scale distributed services

Modern service-oriented applications demand the ability to adapt to changing conditions and unexpected situations while maintaining a required QoS. Existing self-adaptation approaches seem inadequate to address this challenge because many of their assumptions are not met on the large-scale, highly dynamic infrastructures where these applications are generally deployed on. The main motivation of our research is to devise principles that guide the construction of large scale self-adaptive distributed services. We aim to provide sound modeling abstractions based on a clear conceptual background, and their realization as a middleware framework that supports the development of such services. Taking the inspiration from the concepts of decentralized markets in economics, we propose a solution based on three principles: emergent self-organization, utility driven behavior and model-less adaptation. Based on these principles, we designed Collectives, a middleware framework which provides a comprehensive solution for the diverse adaptation concerns that rise in the development of distributed systems. We tested the soundness and comprehensiveness of the Collectives framework by implementing eUDON, a middleware for self-adaptive web services, which we then evaluated extensively by means of a simulation model to analyze its adaptation capabilities in diverse settings. We found that eUDON exhibits the intended properties: it adapts to diverse conditions like peaks in the workload and massive failures, maintaining its QoS and using efficiently the available resources; it is highly scalable and robust; can be implemented on existing services in a non-intrusive way; and do not require any performance model of the services, their workload or the resources they use. We can conclude that our work proposes a solution for the requirements of self-adaptation in demanding usage scenarios without introducing additional complexity. In that sense, we believe we make a significant contribution towards the development of future generation service-oriented applications.Las Aplicaciones Orientadas a Servicios modernas demandan la capacidad de adaptarse a condiciones variables y situaciones inesperadas mientras mantienen un cierto nivel de servio esperado (QoS). Los enfoques de auto-adaptación existentes parecen no ser adacuados debido a sus supuestos no se cumplen en infrastructuras compartidas de gran escala. La principal motivación de nuestra investigación es inerir un conjunto de principios para guiar el desarrollo de servicios auto-adaptativos de gran escala. Nuesto objetivo es proveer abstraciones de modelaje apropiadas, basadas en un marco conceptual claro, y su implemetnacion en un middleware que soporte el desarrollo de estos servicios. Tomando como inspiración conceptos económicos de mercados decentralizados, hemos propuesto una solución basada en tres principios: auto-organización emergente, comportamiento guiado por la utilidad y adaptación sin modelos. Basados en estos principios diseñamos Collectives, un middleware que proveer una solución exhaustiva para los diversos aspectos de adaptación que surgen en el desarrollo de sistemas distribuidos. La adecuación y completitud de Collectives ha sido provada por medio de la implementación de eUDON, un middleware para servicios auto-adaptativos, el ha sido evaluado de manera exhaustiva por medio de un modelo de simulación, analizando sus propiedades de adaptación en diversos escenarios de uso. Hemos encontrado que eUDON exhibe las propiedades esperadas: se adapta a diversas condiciones como picos en la carga de trabajo o fallos masivos, mateniendo su calidad de servicio y haciendo un uso eficiente de los recusos disponibles. Es altamente escalable y robusto; puedeoo ser implementado en servicios existentes de manera no intrusiva; y no requiere la obtención de un modelo de desempeño para los servicios. Podemos concluir que nuestro trabajo nos ha permitido desarrollar una solucion que aborda los requerimientos de auto-adaptacion en escenarios de uso exigentes sin introducir complejidad adicional. En este sentido, consideramos que nuestra propuesta hace una contribución significativa hacia el desarrollo de la futura generación de aplicaciones orientadas a servicios.Postprint (published version

Verfahren und Protokolle für energiebewusste, gemeinsame Ressourcenverwendung mit mobilen Geräten

Voraussetzung für die gemeinsame Nutzung von Betriebsmitteln mobiler Geräte sind Verfahren, die es erlauben, den Energiebedarf für die Bereitstellung von Ressourcen abzuschätzen, unter mehreren Ressourcenanbietern einen geeigneten auszuwählen und diesen für den erbrachten Energieaufwand zu entschädigen. Diese Arbeit stellt hierfür entwickelte Verfahren vor, die insbesondere das geringe Energiebudget mobiler Geräte berücksichtigen

Hochdynamische logische Netzwerke als Infrastruktur mobiler Agentensysteme

Zur sinnvollen Planung ihre Route durch das Internet benötigen mobile Agenten aktuelle Dienstinformationen aus ihrer global verteilten Infrastruktur. Heutige Netzwerke sind allerdings durch die Verwendung von mobilen Endgeräten hochdynamisch geworden, daher skaliert die Frequenz und die Anzahl von nötigen Aktualisierungsnachrichten in globalen logischen Netzwerken nicht mit der Menge der möglichen Dienstangebote. Diese Arbeit konzentriert sich auf die Verteilung, Aktualität und Skalierbarkeit von Dienstinformationen in einem globalen logischen Netzwerk unter Berücksichtigung der Zeitschranken, die von mobilen Endgeräten hervorgerufen werden. Die präsentierte Lösung besteht aus einer Architektur aus drei Infrastrukturdiensten, deren Referenzimplementierung das Infrastrukturdienst-Framework QuickLinkNet darstellt. Dieses baut ein transparentes, zweischichtiges logisches Netzwerk auf, welches als serviceorientierte Infrastruktur für mobile Agentensysteme dient, aber auch für andere verteilte Anwendungen in hochdynamischen Netzwerken verwendet werden kann.In so-called level 2 mobile agent systems, mobile agents are able to plan their route through that network themselves, based on relevant service descriptions and by using dedicated routing services. But today's networks are highly dynamic, since people use mobile devices, and very large in scale. Current infrastructure frameworks are not scaling with the potential number of service descriptions, cannot map the dynamics in the network, and have severe problems with the frequency and number of messages necessary to maintain such an infrastructure. The thesis presented in this paper focuses on exactly this problem of conflicting goals, trying to map scalability (quantity) and flexibility (dynamics) into a unified solution framework. Thereby, timing bounds caused by using mobile devices are the major focus of attention. This leads to a software architecture that comprises of three components. QuickLinkNet implements the achitecture and forms a framework of components, consists of QuickLink, Service-Juggler, and APPLICOOVER. The QuickLinkNet components establish a transparent, logical two-tiered network which can be used as a service-oriented infrastructure for mobile agent systems of level 2 and for other distributed applications in highly dynamic networks as well

Cost-effective resource management for distributed computing

Current distributed computing and resource management infrastructures (e.g., Cluster and Grid) suffer from a wide variety of problems related to resource management, which include scalability bottleneck, resource allocation delay, limited quality-of-service (QoS) support, and lack of cost-aware and service level agreement (SLA) mechanisms. This thesis addresses these issues by presenting a cost-effective resource management solution which introduces the possibility of managing geographically distributed resources in resource units that are under the control of a Virtual Authority (VA). A VA is a collection of resources controlled, but not necessarily owned, by a group of users or an authority representing a group of users. It leverages the fact that different resources in disparate locations will have varying usage levels. By creating smaller divisions of resources called VAs, users would be given the opportunity to choose between a variety of cost models, and each VA could rent resources from resource providers when necessary, or could potentially rent out its own resources when underloaded. The resource management is simplified since the user and owner of a resource recognize only the VA because all permissions and charges are associated directly with the VA. The VA is controlled by a ’rental’ policy which is supported by a pool of resources that the system may rent from external resource providers. As far as scheduling is concerned, the VA is independent from competitors and can instead concentrate on managing its own resources. As a result, the VA offers scalable resource management with minimal infrastructure and operating costs. We demonstrate the feasibility of the VA through both a practical implementation of the prototype system and an illustration of its quantitative advantages through the use of extensive simulations. First, the VA concept is demonstrated through a practical implementation of the prototype system. Further, we perform a cost-benefit analysis of current distributed resource infrastructures to demonstrate the potential cost benefit of such a VA system. We then propose a costing model for evaluating the cost effectiveness of the VA approach by using an economic approach that captures revenues generated from applications and expenses incurred from renting resources. Based on our costing methodology, we present rental policies that can potentially offer effective mechanisms for running distributed and parallel applications without a heavy upfront investment and without the cost of maintaining idle resources. By using real workload trace data, we test the effectiveness of our proposed rental approaches. Finally, we propose an extension to the VA framework that promotes long-term negotiations and rentals based on service level agreements or long-term contracts. Based on the extended framework, we present new SLA-aware policies and evaluate them using real workload traces to demonstrate their effectiveness in improving rental decisions

Redes em malha sem fios baseadas em contexto

Doutoramento em Engenharia ElectrotécnicaIn the modern society, new devices, applications and technologies, with sophisticated capabilities, are converging in the same network infrastructure. Users are also increasingly demanding in personal preferences and expectations, desiring Internet connectivity anytime and everywhere. These aspects have triggered many research efforts, since the current Internet is reaching a breaking point trying to provide enough flexibility for users and profits for operators, while dealing with the complex requirements raised by the recent evolution. Fully aligned with the future Internet research, many solutions have been proposed to enhance the current Internet-based architectures and protocols, in order to become context-aware, that is, to be dynamically adapted to the change of the information characterizing any network entity. In this sense, the presented Thesis proposes a new architecture that allows to create several networks with different characteristics according to their context, on the top of a single Wireless Mesh Network (WMN), which infrastructure and protocols are very flexible and self-adaptable. More specifically, this Thesis models the context of users, which can span from their security, cost and mobility preferences, devices’ capabilities or services’ quality requirements, in order to turn a WMN into a set of logical networks. Each logical network is configured to meet a set of user context needs (for instance, support of high mobility and low security). To implement this user-centric architecture, this Thesis uses the network virtualization, which has often been advocated as a mean to deploy independent network architectures and services towards the future Internet, while allowing a dynamic resource management. This way, network virtualization can allow a flexible and programmable configuration of a WMN, in order to be shared by multiple logical networks (or virtual networks - VNs). Moreover, the high level of isolation introduced by network virtualization can be used to differentiate the protocols and mechanisms of each context-aware VN. This architecture raises several challenges to control and manage the VNs on-demand, in response to user and WMN dynamics. In this context, we target the mechanisms to: (i) discover and select the VN to assign to an user; (ii) create, adapt and remove the VN topologies and routes. We also explore how the rate of variation of the user context requirements can be considered to improve the performance and reduce the complexity of the VN control and management. Finally, due to the scalability limitations of centralized control solutions, we propose a mechanism to distribute the control functionalities along the architectural entities, which can cooperate to control and manage the VNs in a distributed way.Na sociedade actual, novos dispositivos, aplicações e tecnologias, com capacidades sofisticadas, estão a convergir na mesma infra-estrutura de rede. Os utilizadores são também cada vez mais exigentes nas suas preferências e expectativas pessoais, desejando conetividade `a Internet em qualquer hora e lugar. Estes aspectos têm desencadeado muitos esforços de investigação, dado que a Internet atual está a atingir um ponto de rutura ao tentar promover flexibilidade para os utilizadores e lucros para os operadores, enquanto lida com as exigências complexas associadas `a recente evolução. Em sintonia com a linha de investigação para a Internet do futuro, muitas soluções têm sido propostas para melhorar as arquiteturas e protocolos da Internet atual, de forma a torná-los sensíveis ao contexto, isto é, adaptá-los dinamicamente `a alteração da informação que caracteriza qualquer entidade de rede. Neste sentido, a presente Tese propõe uma nova arquitetura que permite criar várias redes com diferentes características de acordo com o contexto das mesmas, sobre uma única rede em malha sem fios (WMN), cuja infra-estructura e protocolos são muito flexíveis e auto-adaptáveis. Mais especificamente, esta Tese modela o contexto dos utilizadores, que pode abranger as suas preferências de segurança, custo e mobilidade, capacidades dos seus dispositivos ou requisitos de qualidade dos seus serviços, de forma a transformar uma WMN num conjunto de redes lógicas. Cada rede lógica ´e configurada para satisfazer um conjunto de necessidades de contexto do utilizador (como exemplo, suporte de mobilidade elevada e de baixa seguran¸ca). Para implementar esta arquitetura centrada no utilizador, esta Tese utiliza a virtualização de redes, que tem muitas vezes sido defendida como um meio para implementar arquiteturas e serviços de rede de uma forma independente, enquanto permite uma gestão dinâmica dos recursos. Desta forma, a virtualização de redes pode permitir uma configuração flexível e programável de uma WMN, a fim de ser partilhada por várias redes lógicas (ou redes virtuais - VNs). Além disso, o grau de isolamento introduzido pela virtualização de redes pode ser utilizado para diferenciar os protocolos e mecanismos de cada VN baseada em contexto. Esta arquitetura levanta vários desafios para controlar e gerir as VNs em tempo real, e em resposta `a dinâmica dos utilizadores e da WMN. Neste contexto, abordamos os mecanismos para: (i) descobrir e selecionar a VN a atribuir a um utilizador; (ii) criar, adaptar e remover as topologias e rotas das VNs. Também exploramos a possibilidade de considerar a taxa de variação dos requisitos de contexto dos utilizadores de forma a melhorar o desempenho e reduzir a complexidade do controlo e gestão das VNs. Finalmente, devido ´as limitações de escalabilidade das soluções de controlo centralizadas, propomos um mecanismo para distribuir as funcionalidades de controlo ao longo das entidades da arquitectura, que podem cooperar para controlar e gerir as VNs de uma forma distribuída

An overlay network for resource discovery in Grids

As Grids try to achieve optimal and balanced utilization of unused resources in a distributed system, fast and efficient discovery of resource states is a key requirement. For small to medium scale Grids, solutions such as the approach in Globus work fine. However, for large, up to global-scale Grids, this approach is not efficient and does not scale. Additionally, even for smaller Grids, a centralized solution will always be a performance bottleneck and a single point of failure. In this paper we investigate the applicability of a structured peer-to-peer system (overlay network) for the discovery of Grid resources. Each node in the Grid becomes a peer in the overlay network, which provides a distributed directory service that allows the participants to discover resources and maintain resource states. Overlay networks implicitly balance load, scale well to very large numbers in terms of nodes and data, and meet the partial failure property of distributed systems, i.e., the system remains operational despite partial failures. We will outline a proof-of-concept implementation based on our P-Grid overlay network, present experimental results from a large-scale deployment on PlanetLab and discuss the pros and cons of overlay networks in the context of Grids