A Taxonomy of Data Grids for Distributed Data Sharing, Management and Processing

Data Grids have been adopted as the platform for scientific communities that need to share, access, transport, process and manage large data collections distributed worldwide. They combine high-end computing technologies with high-performance networking and wide-area storage management techniques. In this paper, we discuss the key concepts behind Data Grids and compare them with other data sharing and distribution paradigms such as content delivery networks, peer-to-peer networks and distributed databases. We then provide comprehensive taxonomies that cover various aspects of architecture, data transportation, data replication and resource allocation and scheduling. Finally, we map the proposed taxonomy to various Data Grid systems not only to validate the taxonomy but also to identify areas for future exploration. Through this taxonomy, we aim to categorise existing systems to better understand their goals and their methodology. This would help evaluate their applicability for solving similar problems. This taxonomy also provides a "gap analysis" of this area through which researchers can potentially identify new issues for investigation. Finally, we hope that the proposed taxonomy and mapping also helps to provide an easy way for new practitioners to understand this complex area of research.Comment: 46 pages, 16 figures, Technical Repor

The role of intelligent systems in the development of peer-to-peer systems for energetic distribution management

Intelligent Systems are one of today’s greatest strengths, while environmental sustainability is one of today’s biggest challenges. This study aims to integrate the most recent innovations in intelligent technologies with the development of smart energy grids and Peer-to-Peer (P2P) systems for energetic distribution. Specifically, it investigates the complex relations between these concepts, while analysing how developments in each field both influence and are influenced by each other. To do so, this study answers three research questions. The first one regards the implementation of Intelligent Systems, the second concerns the development of Smart Grids, while the third is concerned with the possibility of building P2P Systems. To provide relevant conclusions, an extensive literature review regarding all subjects was carried, along with a statistical analysis of three online surveys. The obtained results show that there are significant influences and connections between the development of intelligent technologies and the implementation of smart grids and P2P Systems, thus supporting several hypotheses formulated for this study. On this basis, conclusions are drawn concerning the high value of each topic in separate, and the even higher value of the topics when integrated.Sistemas Inteligentes são um dos maiores benefícios dos dias de hoje, enquanto a sustentabilidade ambiental é um dos maiores desafios. Este estudo pretende integrar as mais recentes inovações em tecnologias inteligentes com o desenvolvimento de redes de energia inteligentes (Smart Grids) e sistemas Peer-to-Peer (P2P) para distribuição energética. Especificamente, investiga as relações complexas entre estes conceitos, enquanto analisa como desenvolvimentos em cada área influenciam e são influenciados pelas outras. Para isso, este estudo responde a três questões de pesquisa. A primeira relaciona-se com a implementação de Sistemas Inteligentes, a segunda com o desenvolvimento de Redes Inteligentes, e a terceira está relacionada com a possibilidade de construir Sistemas P2P. Para obter conclusões relevantes, foi feita uma extensa revisão de literatura relativa a todos os temas, assim como uma análise estatística de três questionários online. Os resultados obtidos mostram que existem influências e conexões significativas entre o desenvolvimento de tecnologias inteligentes e a implementação de Smart Grids e Sistemas P2P, suportando assim múltiplas hipóteses formuladas para este estudo. Com esta base, são retiradas conclusões que confirmam o elevado valor de cada tópico em separado, e o ainda maior valor dos tópicos quando integrados

Service-Oriented Ad Hoc Grid Computing

Subject of this thesis are the design and implementation of an ad hoc Grid infrastructure. The vision of an ad hoc Grid further evolves conventional service-oriented Grid systems into a more robust, more flexible and more usable environment that is still standards compliant and interoperable with other Grid systems. A lot of work in current Grid middleware systems is focused on providing transparent access to high performance computing (HPC) resources (e.g. clusters) in virtual organizations spanning multiple institutions. The ad hoc Grid vision presented in this thesis exceeds this view in combining classical Grid components with more flexible components and usage models, allowing to form an environment combining dedicated HPC-resources with a large number of personal computers forming a "Desktop Grid". Three examples from medical research, media research and mechanical engineering are presented as application scenarios for a service-oriented ad hoc Grid infrastructure. These sample applications are also used to derive requirements for the runtime environment as well as development tools for such an ad hoc Grid environment. These requirements form the basis for the design and implementation of the Marburg ad hoc Grid Environment (MAGE) and the Grid Development Tools for Eclipse (GDT). MAGE is an implementation of a WSRF-compliant Grid middleware, that satisfies the criteria for an ad hoc Grid middleware presented in the introduction to this thesis. GDT extends the popular Eclipse integrated development environment by components that support application development both for traditional service-oriented Grid middleware systems as well as ad hoc Grid infrastructures such as MAGE. These development tools represent the first fully model driven approach to Grid service development integrated with infrastructure management components in service-oriented Grid computing. This thesis is concluded by a quantitative discussion of the performance overhead imposed by the presented extensions to a service-oriented Grid middleware as well as a discussion of the qualitative improvements gained by the overall solution. The conclusion of this thesis also gives an outlook on future developments and areas for further research. One of these qualitative improvements is "hot deployment" the ability to install and remove Grid services in a running node without interrupt to other active services on the same node. Hot deployment has been introduced as a novelty in service-oriented Grid systems as a result of the research conducted for this thesis. It extends service-oriented Grid computing with a new paradigm, making installation of individual application components a functional aspect of the application. This thesis further explores the idea of using peer-to-peer (P2P networking for Grid computing by combining a general purpose P2P framework with a standard compliant Grid middleware. In previous work the application of P2P systems has been limited to replica location and use of P2P index structures for discovery purposes. The work presented in this thesis also uses P2P networking to realize seamless communication accross network barriers. Even though the web service standards have been designed for the internet, the two-way communication requirement introduced by the WSRF-standards and particularly the notification pattern is not well supported by the web service standards. This defficiency can be answered by mechanisms that are part of such general purpose P2P communication frameworks. Existing security infrastructures for Grid systems focus on protection of data during transmission and access control to individual resources or the overall Grid environment. This thesis focuses on security issues within a single node of a dynamically changing service-oriented Grid environment. To counter the security threads arising from the new capabilities of an ad hoc Grid, a number of novel isolation solutions are presented. These solutions address security issues and isolation on a fine-grained level providing a range of applicable basic mechanisms for isolation, ranging from lightweight system call interposition to complete para-virtualization of the operating systems