Data Management in the APPA System

International audienceCombining Grid and P2P technologies can be exploited to provide high-level data sharing in large-scale distributed environments. However, this combination must deal with two hard problems: the scale of the network and the dynamic behavior of the nodes. In this paper, we present our solution in APPA (Atlas Peer-to-Peer Architecture), a data management system with high-level services for building large-scale distributed applications. We focus on data availability and data discovery which are two main requirements for implementing large-scale Grids. We have validated APPA's services through a combination of experimentation over Grid5000, which is a very large Grid experimental platform, and simulation using SimJava. The results show very good performance in terms of communication cost and response time

Global Grids and Software Toolkits: A Study of Four Grid Middleware Technologies

Grid is an infrastructure that involves the integrated and collaborative use of computers, networks, databases and scientific instruments owned and managed by multiple organizations. Grid applications often involve large amounts of data and/or computing resources that require secure resource sharing across organizational boundaries. This makes Grid application management and deployment a complex undertaking. Grid middlewares provide users with seamless computing ability and uniform access to resources in the heterogeneous Grid environment. Several software toolkits and systems have been developed, most of which are results of academic research projects, all over the world. This chapter will focus on four of these middlewares--UNICORE, Globus, Legion and Gridbus. It also presents our implementation of a resource broker for UNICORE as this functionality was not supported in it. A comparison of these systems on the basis of the architecture, implementation model and several other features is included.Comment: 19 pages, 10 figure

Découverte et allocation des ressources pour le traitement de requêtes dans les systèmes grilles

De nos jours, les systèmes Grille, grâce à leur importante capacité de calcul et de stockage ainsi que leur disponibilité, constituent l'un des plus intéressants environnements informatiques. Dans beaucoup de différents domaines, on constate l'utilisation fréquente des facilités que les environnements Grille procurent. Le traitement des requêtes distribuées est l'un de ces domaines où il existe de grandes activités de recherche en cours, pour transférer l'environnement sous-jacent des systèmes distribués et parallèles à l'environnement Grille. Dans le cadre de cette thèse, nous nous concentrons sur la découverte des ressources et des algorithmes d'allocation de ressources pour le traitement des requêtes dans les environnements Grille. Pour ce faire, nous proposons un algorithme de découverte des ressources pour le traitement des requêtes dans les systèmes Grille en introduisant le contrôle de topologie auto-stabilisant et l'algorithme de découverte des ressources dirigé par l'élection convergente. Ensuite, nous présentons un algorithme d'allocation des ressources, qui réalise l'allocation des ressources pour les requêtes d'opérateur de jointure simple par la génération d'un espace de recherche réduit pour les nœuds candidats et en tenant compte des proximités des candidats aux sources de données. Nous présentons également un autre algorithme d'allocation des ressources pour les requêtes d'opérateurs de jointure multiple. Enfin, on propose un algorithme d'allocation de ressources, qui apporte une tolérance aux pannes lors de l'exécution de la requête par l'utilisation de la réplication passive d'opérateurs à état. La contribution générale de cette thèse est double. Premièrement, nous proposons un nouvel algorithme de découverte de ressource en tenant compte des caractéristiques des environnements Grille. Nous nous adressons également aux problèmes d'extensibilité et de dynamicité en construisant une topologie efficace sur l'environnement Grille et en utilisant le concept d'auto-stabilisation, et par la suite nous adressons le problème de l'hétérogénéité en proposant l'algorithme de découverte de ressources dirigé par l'élection convergente. La deuxième contribution de cette thèse est la proposition d'un nouvel algorithme d'allocation des ressources en tenant compte des caractéristiques de l'environnement Grille. Nous abordons les problèmes causés par la grande échelle caractéristique en réduisant l'espace de recherche pour les ressources candidats. De ce fait nous réduisons les coûts de communication au cours de l'exécution de la requête en allouant des nœuds au plus près des sources de données. Et enfin nous traitons la dynamicité des nœuds, du point de vue de leur existence dans le système, en proposant un algorithme d'affectation des ressources avec une tolérance aux pannes.Grid systems are today's one of the most interesting computing environments because of their large computing and storage capabilities and their availability. Many different domains profit the facilities of grid environments. Distributed query processing is one of these domains in which there exists large amounts of ongoing research to port the underlying environment from distributed and parallel systems to the grid environment. In this thesis, we focus on resource discovery and resource allocation algorithms for query processing in grid environments. For this, we propose resource discovery algorithm for query processing in grid systems by introducing self-stabilizing topology control and converge-cast based resource discovery algorithms. Then, we propose a resource allocation algorithm, which realizes allocation of resources for single join operator queries by generating a reduced search space for the candidate nodes and by considering proximities of candidates to the data sources. We also propose another resource allocation algorithm for queries with multiple join operators. Lastly, we propose a fault-tolerant resource allocation algorithm, which provides fault-tolerance during the execution of the query by the use of passive replication of stateful operators. The general contribution of this thesis is twofold. First, we propose a new resource discovery algorithm by considering the characteristics of the grid environments. We address scalability and dynamicity problems by constructing an efficient topology over the grid environment using the self-stabilization concept; and we deal with the heterogeneity problem by proposing the converge-cast based resource discovery algorithm. The second main contribution of this thesis is the proposition of a new resource allocation algorithm considering the characteristics of the grid environment. We tackle the scalability problem by reducing the search space for candidate resources. We decrease the communication costs during the query execution by allocating nodes closer to the data sources. And finally we deal with the dynamicity of nodes, in terms of their existence in the system, by proposing the fault-tolerant resource allocation algorithm

A Taxonomy of Workflow Management Systems for Grid Computing

With the advent of Grid and application technologies, scientists and engineers are building more and more complex applications to manage and process large data sets, and execute scientific experiments on distributed resources. Such application scenarios require means for composing and executing complex workflows. Therefore, many efforts have been made towards the development of workflow management systems for Grid computing. In this paper, we propose a taxonomy that characterizes and classifies various approaches for building and executing workflows on Grids. We also survey several representative Grid workflow systems developed by various projects world-wide to demonstrate the comprehensiveness of the taxonomy. The taxonomy not only highlights the design and engineering similarities and differences of state-of-the-art in Grid workflow systems, but also identifies the areas that need further research.Comment: 29 pages, 15 figure

Distributed virtual environment scalability and security

Distributed virtual environments (DVEs) have been an active area of research and engineering for more than 20 years. The most widely deployed DVEs are network games such as Quake, Halo, and World of Warcraft (WoW), with millions of users and billions of dollars in annual revenue. Deployed DVEs remain expensive centralized implementations despite significant research outlining ways to distribute DVE workloads. This dissertation shows previous DVE research evaluations are inconsistent with deployed DVE needs. Assumptions about avatar movement and proximity - fundamental scale factors - do not match WoW’s workload, and likely the workload of other deployed DVEs. Alternate workload models are explored and preliminary conclusions presented. Using realistic workloads it is shown that a fully decentralized DVE cannot be deployed to today’s consumers, regardless of its overhead. Residential broadband speeds are improving, and this limitation will eventually disappear. When it does, appropriate security mechanisms will be a fundamental requirement for technology adoption. A trusted auditing system (“Carbon”) is presented which has good security, scalability, and resource characteristics for decentralized DVEs. When performing exhaustive auditing, Carbon adds 27% network overhead to a decentralized DVE with a WoW-like workload. This resource consumption can be reduced significantly, depending upon the DVE’s risk tolerance. Finally, the Pairwise Random Protocol (PRP) is described. PRP enables adversaries to fairly resolve probabilistic activities, an ability missing from most decentralized DVE security proposals. Thus, this dissertations contribution is to address two of the obstacles for deploying research on decentralized DVE architectures. First, lack of evidence that research results apply to existing DVEs. Second, the lack of security systems combining appropriate security guarantees with acceptable overhead

A novel multi-level and community-based agent ecosystem to support customers dynamic decision-making in smart grids

Electrical systems have evolved at a fast pace over the past years, particularly in response to the current environmental and climate challenges. Consequently, the European Union and the United Nations have encouraged the development of a more sustainable energy strategy. This strategy triggered a paradigm shift in energy consumption and production, which becoming increasingly distributed, resulted in the development and emergence of smart energy grids. Multi-agent systems are one of the most widely used artificial intelligence concepts in smart grids. Both multi-agent systems and smart grids are distributed, so there is correspondence between the used technology and the network's complex reality. Due to the wide variety of multi-agent systems applied to smart grids, which typically have very specific goals, the ability to model the network as a whole may be compromised, as communication between systems is typically non-existent. This dissertation, therefore, proposes an agent-based ecosystem to model smart grids in which different agent-based systems can coexist. This dissertation aims to conceive, implement, test, and validate a new agent-based ecosystem, entitled A4SG (agent-based ecosystem for smart grids modelling), which combines the concepts of multi-agent systems and agent communities to enable the modelling and representation of smart grids and the entities that compose them. The proposed ecosystem employs an innovative methodology for managing static or dynamic interactions present in smart grids. The creation of a solution that allows the integration of existing systems into an ecosystem, enables the representation of smart grids in a realistic and comprehensive manner. A4SG integrates several functionalities that support the ecosystem's management, also conceived, implemented, tested, and validated in this dissertation. Two mobility functionalities are proposed: one that allows agents to move between physical machines and another that allows "virtual" mobility, where agents move between agent communities to improve the context for the achievement of their objectives. In order to prevent an agent from becoming overloaded, a novel functionality is proposed to enable the creation of agents that function as extensions of the main agent (i.e., branch agents), allowing the distribution of objectives among the various extensions of the main agent. Several case studies, which test the proposed services and functionalities individually and the ecosystem as a whole, were used to test and validate the proposed solution. These case studies were conducted in realistic contexts using data from multiple sources, including energy communities. The results indicate that the used methodologies can increase participation in demand response events, increasing the fitting between consumers and aggregators from 12 % to 69 %, and improve the strategies used in energy transaction markets, allowing an energy community of 50 customers to save 77.0 EUR per week.Os últimos anos têm sido de mudança nos sistemas elétricos, especialmente devido aos atuais desafios ambientais e climáticos. A procura por uma estratégia mais sustentável para o domínio da energia tem sido promovida pela União Europeia e pela Organização das Nações Unidas. A mudança de paradigma no que toca ao consumo e produção de energia, que acontece, cada vez mais, de forma distribuída, tem levado à emergência das redes elétricas inteligentes. Os sistemas multi-agente são um dos conceitos, no domínio da inteligência artificial, mais aplicados em redes inteligentes. Tanto os sistemas multi-agente como as redes inteligentes têm uma natureza distribuída, existindo por isso um alinhamento entre a tecnologia usada e a realidade complexa da rede. Devido a existir uma vasta oferta de sistemas multi-agente aplicados a redes inteligentes, normalmente com objetivos bastante específicos, a capacidade de modelar a rede como um todo pode ficar comprometida, porque a comunicação entre sistemas é, geralmente, inexistente. Por isso, esta dissertação propõe um ecossistema baseado em agentes para modelar as redes inteligentes, onde vários sistemas de agentes coexistem. Esta dissertação pretende conceber, implementar, testar, e validar um novo ecossistema multiagente, intitulado A4SG (agent-based ecosystem for smart grids modelling), que combina os conceitos de sistemas multi-agente e comunidades de agentes, permitindo a modelação e representação de redes inteligentes e das suas entidades. O ecossistema proposto utiliza uma metodologia inovadora para gerir as interações presentes nas redes inteligentes, sejam elas estáticas ou dinâmicas. A criação de um ecossistema que permite a integração de sistemas já existentes, cria a possibilidade de uma representação realista e detalhada das redes de energia. O A4SG integra diversas funcionalidades, também estas concebidas, implementadas, testadas, e validadas nesta dissertação, que suportam a gestão do próprio ecossistema. São propostas duas funcionalidades de mobilidade, uma que permite aos agentes mover-se entre máquinas físicas, e uma que permite uma mobilidade “virtual”, onde os agentes se movem entre comunidades de agentes, de forma a melhorar o contexto para a execução dos seus objetivos. É também proposta uma nova funcionalidade que permite a criação de agentes que funcionam como uma extensão de um agente principal, com o objetivo de evitar a sobrecarga de um agente, permitindo a distribuição de objetivos entre as várias extensões do agente principal. A solução proposta foi testada e validada por vários casos de estudo, que testam os serviços e funcionalidades propostas individualmente, e o ecossistema como um todo. Estes casos de estudo foram executados em contextos realistas, usando dados provenientes de diversas fontes, tais como comunidades de energia. Os resultados demonstram que as metodologias utilizadas podem melhorar a participação em eventos de demand response, subindo a adequação entre consumidores e agregadores de 12 % para 69 %, e melhorar as estratégias utilizadas em mercados de transações de energia, permitindo a uma comunidade de energia com 50 consumidores poupar 77,0 EUR por semana