A Taxonomy of Data Grids for Distributed Data Sharing, Management and Processing

Data Grids have been adopted as the platform for scientific communities that need to share, access, transport, process and manage large data collections distributed worldwide. They combine high-end computing technologies with high-performance networking and wide-area storage management techniques. In this paper, we discuss the key concepts behind Data Grids and compare them with other data sharing and distribution paradigms such as content delivery networks, peer-to-peer networks and distributed databases. We then provide comprehensive taxonomies that cover various aspects of architecture, data transportation, data replication and resource allocation and scheduling. Finally, we map the proposed taxonomy to various Data Grid systems not only to validate the taxonomy but also to identify areas for future exploration. Through this taxonomy, we aim to categorise existing systems to better understand their goals and their methodology. This would help evaluate their applicability for solving similar problems. This taxonomy also provides a "gap analysis" of this area through which researchers can potentially identify new issues for investigation. Finally, we hope that the proposed taxonomy and mapping also helps to provide an easy way for new practitioners to understand this complex area of research.Comment: 46 pages, 16 figures, Technical Repor

Symbiosis between the TRECVid benchmark and video libraries at the Netherlands Institute for Sound and Vision

Audiovisual archives are investing in large-scale digitisation efforts of their analogue holdings and, in parallel, ingesting an ever-increasing amount of born- digital files in their digital storage facilities. Digitisation opens up new access paradigms and boosted re-use of audiovisual content. Query-log analyses show the shortcomings of manual annotation, therefore archives are complementing these annotations by developing novel search engines that automatically extract information from both audio and the visual tracks. Over the past few years, the TRECVid benchmark has developed a novel relationship with the Netherlands Institute of Sound and Vision (NISV) which goes beyond the NISV just providing data and use cases to TRECVid. Prototype and demonstrator systems developed as part of TRECVid are set to become a key driver in improving the quality of search engines at the NISV and will ultimately help other audiovisual archives to offer more efficient and more fine-grained access to their collections. This paper reports the experiences of NISV in leveraging the activities of the TRECVid benchmark

QoS Contract Negotiation in Distributed Component-Based Software

Currently, several mature and commercial component models (for e.g. EJB, .NET, COM+) exist on the market. These technologies were designed largely for applications with business-oriented non-functional requirements such as data persistence, confidentiality, and transactional support. They provide only limited support for the development of components and applications with non-functional properties (NFPs) like QoS (e.g. throughput, response time). The integration of QoS into component infrastructure requires among other things the support of components’ QoS contract specification, negotiation, adaptation, etc. This thesis focuses on contract negotiation. For applications in which the consideration of non-functional properties (NFPs) is essential (e.g. Video-on-Demand, eCommerce), a component-based solution demands the appropriate composition of the QoS contracts specified at the different ports of the collaborating components. The ports must be properly connected so that the QoS level required by one is matched by the QoS level provided by the other. Generally, QoS contracts of components depend on run-time resources (e.g. network bandwidth, CPU time) or quality attributes to be established dynamically and are usually specified in multiple QoS-Profiles. QoS contract negotiation enables the selection of appropriate concrete QoS contracts between collaborating components. In our approach, the component containers perform the contract negotiation at run-time. This thesis addresses the QoS contract negotiation problem by first modelling it as a constraint satisfaction optimization problem (CSOP). As a basis for this modelling, the provided and required QoS as well as resource demand are specified at the component level. The notion of utility is applied to select a good solution according to some negotiation goal (e.g. user’s satisfaction). We argue that performing QoS contract negotiation in multiple phases simplifies the negotiation process and makes it more efficient. Based on such classification, the thesis presents heuristic algorithms that comprise coarse-grained and fine-grained negotiations for collaborating components deployed in distributed nodes in the following scenarios: (i) single-client - single-server, (ii) multiple-clients, and (iii) multi-tier scenarios. To motivate the problem as well as to validate the proposed approach, we have examined three componentized distributed applications. These are: (i) video streaming, (ii) stock quote, and (iii) billing (to evaluate certain security properties). An experiment has been conducted to specify the QoS contracts of the collaborating components in one of the applications we studied. In a run-time system that implements our algorithm, we simulated different behaviors concerning: (i) user’s QoS requirements and preferences, (ii) resource availability conditions concerning the client, server, and network bandwidth, and (iii) the specified QoS-Profiles of the collaborating components. Under various conditions, the outcome of the negotiation confirms the claim we made with regard to obtaining a good solution

QoS Contract Negotiation in Distributed Component-Based Software

Currently, several mature and commercial component models (for e.g. EJB, .NET, COM+) exist on the market. These technologies were designed largely for applications with business-oriented non-functional requirements such as data persistence, confidentiality, and transactional support. They provide only limited support for the development of components and applications with non-functional properties (NFPs) like QoS (e.g. throughput, response time). The integration of QoS into component infrastructure requires among other things the support of components’ QoS contract specification, negotiation, adaptation, etc. This thesis focuses on contract negotiation. For applications in which the consideration of non-functional properties (NFPs) is essential (e.g. Video-on-Demand, eCommerce), a component-based solution demands the appropriate composition of the QoS contracts specified at the different ports of the collaborating components. The ports must be properly connected so that the QoS level required by one is matched by the QoS level provided by the other. Generally, QoS contracts of components depend on run-time resources (e.g. network bandwidth, CPU time) or quality attributes to be established dynamically and are usually specified in multiple QoS-Profiles. QoS contract negotiation enables the selection of appropriate concrete QoS contracts between collaborating components. In our approach, the component containers perform the contract negotiation at run-time. This thesis addresses the QoS contract negotiation problem by first modelling it as a constraint satisfaction optimization problem (CSOP). As a basis for this modelling, the provided and required QoS as well as resource demand are specified at the component level. The notion of utility is applied to select a good solution according to some negotiation goal (e.g. user’s satisfaction). We argue that performing QoS contract negotiation in multiple phases simplifies the negotiation process and makes it more efficient. Based on such classification, the thesis presents heuristic algorithms that comprise coarse-grained and fine-grained negotiations for collaborating components deployed in distributed nodes in the following scenarios: (i) single-client - single-server, (ii) multiple-clients, and (iii) multi-tier scenarios. To motivate the problem as well as to validate the proposed approach, we have examined three componentized distributed applications. These are: (i) video streaming, (ii) stock quote, and (iii) billing (to evaluate certain security properties). An experiment has been conducted to specify the QoS contracts of the collaborating components in one of the applications we studied. In a run-time system that implements our algorithm, we simulated different behaviors concerning: (i) user’s QoS requirements and preferences, (ii) resource availability conditions concerning the client, server, and network bandwidth, and (iii) the specified QoS-Profiles of the collaborating components. Under various conditions, the outcome of the negotiation confirms the claim we made with regard to obtaining a good solution

Performance Problem Diagnostics by Systematic Experimentation

Diagnostics of performance problems requires deep expertise in performance engineering and entails a high manual effort. As a consequence, performance evaluations are postponed to the last minute of the development process. In this thesis, we introduce an automatic, experiment-based approach for performance problem diagnostics in enterprise software systems. With this approach, performance engineers can concentrate on their core competences instead of conducting repeating tasks

Performance Problem Diagnostics by Systematic Experimentation

In this book, we introduce an automatic, experiment-based approach for performance problem diagnostics in enterprise software systems. The proposed approach systematically searches for root causes of detected performance problems by executing series of systematic performance tests. The presented approach is evaluated by various case studies showing that the presented approach is applicable to a wide range of contexts

Adaptive Caching of Distributed Components

Die Zugriffslokalität referenzierter Daten ist eine wichtige Eigenschaft verteilter Anwendungen. Lokales Zwischenspeichern abgefragter entfernter Daten (Caching) wird vielfach bei der Entwicklung solcher Anwendungen eingesetzt, um diese Eigenschaft auszunutzen. Anschliessende Zugriffe auf diese Daten können so beschleunigt werden, indem sie aus dem lokalen Zwischenspeicher bedient werden. Gegenwärtige Middleware-Architekturen bieten dem Anwendungsprogrammierer jedoch kaum Unterstützung für diesen nicht-funktionalen Aspekt. Die vorliegende Arbeit versucht deshalb, Caching als separaten, konfigurierbaren Middleware-Dienst auszulagern. Durch die Einbindung in den Softwareentwicklungsprozess wird die frühzeitige Modellierung und spätere Wiederverwendung caching-spezifischer Metadaten gewährleistet. Zur Laufzeit kann sich das entwickelte System außerdem bezüglich der Cachebarkeit von Daten adaptiv an geändertes Nutzungsverhalten anpassen.Locality of reference is an important property of distributed applications. Caching is typically employed during the development of such applications to exploit this property by locally storing queried data: Subsequent accesses can be accelerated by serving their results immediately form the local store. Current middleware architectures however hardly support this non-functional aspect. The thesis at hand thus tries outsource caching as a separate, configurable middleware service. Integration into the software development lifecycle provides for early capturing, modeling, and later reuse of cachingrelated metadata. At runtime, the implemented system can adapt to caching access characteristics with respect to data cacheability properties, thus healing misconfigurations and optimizing itself to an appropriate configuration. Speculative prefetching of data probably queried in the immediate future complements the presented approach