Methods and design issues for next generation network-aware applications

Networks are becoming an essential component of modern cyberinfrastructure and this work describes methods of designing distributed applications for high-speed networks to improve application scalability, performance and capabilities. As the amount of data generated by scientific applications continues to grow, to be able to handle and process it, applications should be designed to use parallel, distributed resources and high-speed networks. For scalable application design developers should move away from the current component-based approach and implement instead an integrated, non-layered architecture where applications can use specialized low-level interfaces. The main focus of this research is on interactive, collaborative visualization of large datasets. This work describes how a visualization application can be improved through using distributed resources and high-speed network links to interactively visualize tens of gigabytes of data and handle terabyte datasets while maintaining high quality. The application supports interactive frame rates, high resolution, collaborative visualization and sustains remote I/O bandwidths of several Gbps (up to 30 times faster than local I/O). Motivated by the distributed visualization application, this work also researches remote data access systems. Because wide-area networks may have a high latency, the remote I/O system uses an architecture that effectively hides latency. Five remote data access architectures are analyzed and the results show that an architecture that combines bulk and pipeline processing is the best solution for high-throughput remote data access. The resulting system, also supporting high-speed transport protocols and configurable remote operations, is up to 400 times faster than a comparable existing remote data access system. Transport protocols are compared to understand which protocol can best utilize high-speed network connections, concluding that a rate-based protocol is the best solution, being 8 times faster than standard TCP. An HD-based remote teaching application experiment is conducted, illustrating the potential of network-aware applications in a production environment. Future research areas are presented, with emphasis on network-aware optimization, execution and deployment scenarios

Generic Metadata Handling in Scientific Data Life Cycles

Scientific data life cycles define how data is created, handled, accessed, and analyzed by users. Such data life cycles become increasingly sophisticated as the sciences they deal with become more and more demanding and complex with the coming advent of exascale data and computing. The overarching data life cycle management background includes multiple abstraction categories with data sources, data and metadata management, computing and workflow management, security, data sinks, and methods on how to enable utilization. Challenges in this context are manifold. One is to hide the complexity from the user and to enable seamlessness in using resources to usability and efficiency. Another one is to enable generic metadata management that is not restricted to one use case but can be adapted with limited effort to further ones. Metadata management is essential to enable scientists to save time by avoiding the need for manually keeping track of data, meaning for example by its content and location. As the number of files grows into the millions, managing data without metadata becomes increasingly difficult. Thus, the solution is to employ metadata management to enable the organization of data based on information about it. Previously, use cases tended to only support highly specific or no metadata management at all. Now, a generic metadata management concept is available that can be used to efficiently integrate metadata capabilities with use cases. The concept was implemented within the MoSGrid data life cycle that enables molecular simulations on distributed HPC-enabled data and computing infrastructures. The implementation enables easy-to-use and effective metadata management. Automated extraction, annotation, and indexing of metadata was designed, developed, integrated, and search capabilities provided via a seamless user interface. Further analysis runs can be directly started based on search results. A complete evaluation of the concept both in general and along the example implementation is presented. In conclusion, generic metadata management concept advances the state of the art in scientific date life cycle management

Orchestration of resources in distributed, heterogeneous grid environments using dynamic service level agreements

Die Akzeptanz des Internets und der zunehmende Ausbau von Netzwerkkapazitäten ermöglichen bereits heute einen effizienten und zuverlässigen Austausch riesiger Datenmengen zwischen verschiedenen Rechensystemen weltweit. Hieraus resultieren neue Paradigmen bei der Bereit-stellung und Nutzung verteilter IT-Ressourcen wie zum Beispiel das Grid-Computing. Im Grid-Computing werden Rechenressourcen verschiedener Institutionen bzw. Organisationen koordiniert zur Lösung wissenschaftlicher und wirtschaftlicher Problemstellungen genutzt. Neben Rechenressourcen werden dabei auch Daten, Datenspeicher oder Software bereitgestellt. Die Qualität mit der diese Ressourcen bereitgestellt werden gewinnt dabei zunehmend an Bedeutung. Qualitätseigenschaften sind zum Beispiel die minimale Verfügbarkeit von Rechenressourcen, die maximale Zugriffszeit eines Datenspeichers oder die maximale Antwortzeit einer web-basierten Anwendung. Für Ressourcenanbieter bedeutet dies dass spezifische Prozesse implementiert werden müssen um qualitativ hochwertige IT-Dienste bereitzustellen. Zudem können Dienste mit unterschiedlichen Dienstqualitäten bereitgestellt werden, wobei Dienste mit geringerer Qualität preiswerter angeboten werden als solche mit hoher Qualität. Anwender hingegen können den für sie passenden Dienst hinsichtlich ihrer Anforderungen und ihres Budgets auswählen. Service Level Agreements (SLAs) sind ein akzeptierter Ansatz um Verträge über IT-Dienste und Dienstqualitäten zu realisieren. SLAs beschreiben sowohl die funktionalen als auch die nicht-funktionalen Anforderungen von IT-Diensten als auch Vergütung und Strafen für Erfüllung bzw. Nichterfüllung der definierten Anforderungen. Diese Arbeit behandelt Methoden zur Verhandlung und Verwaltung von dynamischen SLAs in verteilten Systemen auf Basis des WS-Agreement Standards. Im Fokus steht hierbei die Deklaration von SLAs, deren automatisierte Verhandlung und Erstellung, das Monitoring von SLA Garantien, sowie die Verwendung von SLAs zur koordinierten Nutzung von IT-Ressourcen. Zu diesem Zweck wurde aufbauend auf die WS-Agreement Spezifikation ein Protokoll zur dynamischen Verhandlung bzw. Neuverhandlung von SLAs entwickelt. Dies beinhaltet die Definition eines Verhandlungsmodells zum Austausch von Angeboten zwischen den Verhandlungspartnern. Die anschließende Erstellung der SLAs basiert auf dem WS-Agreement Standard stellt einen automatisierter Prozess dar. Da es sich bei SLAs um elektronische Verträge handelt wurden Mechanismen zur Validierung von SLA Angeboten entwickelt und im Detail vorgestellt. Darüber hinaus werden Methoden zur automatisierten Evaluation von SLA Garantien beschrieben. Abschließend wird die Architektur und Implementierung eines Orchestrierungsdienstes zur Co-Allokation beliebiger Ressource wie z.B. Rechen- und Netzwerkressourcen vorgestellt. Die Ressourcenorchestrierung wurde hierbei mittels SLAs realisiert.In recent decades the acceptance of the internet and the increase of network capacity have resulted in a situation in which it is now possible to transfer huge amounts of data efficiently and reliably between different computing systems worldwide. This enables new paradigms in provision and use of distributed IT resources. Grid computing is such a well-known paradigm where computing resources owned by various institutions and organizations are used in a coordinated way in order to solve scientific and economic problems. Besides computing resources also data, storage or software resources are provided. Today it becomes more and more important with which quality the different resources are provided. This may be, for example, the minimal availability of computing resources, the maximum access time of a data storage or the maximum response time of a web-based application. Offering resources with a defined quality means for resource providers that they need to implement specific processes to assert the quality of the provisioning process. On the other hand, resource providers can offer their services at different quality levels. Services with a lower quality can be offered cheaper than those with a higher quality. Service consumers can therefore select the service with the appropriate service level in terms of their requirements and budget. This provides both parties, service provider and consumer, with more flexibility during the service provisioning process. Service level agreements (SLAs) are an accepted approach to realize contracts for IT services and service qualities. They describe the functional and the non-functional requirements of IT services. Additionally, they define compensation and penalties for delivering services with the defined requirements respectively for failing to meet these quality criteria. This thesis examines methods for negotiation and management of SLAs in distributed systems based on the WS-Agreement standard. The focus is on methods for SLA declaration, automated SLA negotiation and creation processes, monitoring of SLA guarantees, and the application of SLAs for coordinated IT resource provisioning. Therefore, a protocol for dynamic negotiation or renegotiation of SLAs is developed as an extension to the WS-Agreement specification. This includes the definition of a negotiation model for the exchange of offers between the negotiating partners. The subsequent SLA creation process is an automated process in distributed systems. Since SLAs are a kind of electronic contracts a mechanism for validating the integrity of SLA offers was developed and is presented in detail. In addition, automatic methods for SLA guarantee evaluation are described. Finally, an orchestration service for co-allocating arbitrary resources such as computing and network resources is presented. The resource orchestration process has been realized using SLAs. The architecture of this service is evaluated and based on the evaluation result an advanced orchestration service architecture is conceived

A generic scheduling architecture for service oriented distributed computing infrastructures

In state-of-the-art distributed computing infrastructures different kinds of resources are combined to offer complex services to customers. As of today, service-oriented middleware stacks are the work-horses to connect resources and their users, and to implement all functions needed to provide those services. Analysing the functionality of prominent middleware stacks, it becomes evident that common challenges, like scalability, manageability, efficiency, reliability, security, or complexity, exist, and that they constitute major research areas in information and communication technologies in general and distributed systems in particular. One core issue, touching all of the aforementioned challenges, is the question of how to distribute units of work in a distributed computing infrastructure, a task generally referred to as scheduling. Integrating a variety of resources and services while being compliant with well-defined business objectives makes the development of scheduling strategies and services a difficult venture, which, for service-oriented distributed computing infrastructures, translates to the assignment of services to activities over time aiming at the optimisation of multiple, potentially competing, quality-of-service criteria. Many concepts, methods, and tools for scheduling in distributed computing infrastructures exist, a majority of which being dedicated to provide algorithmic solutions and schedulers. We approach the problem from another angle and offer a more general answer to the question of ’how to design an automated scheduling process and an architecture supporting it’. Doing so, we take special care of the service-oriented nature of the systems we consider and of the integration of our solutions into IT service management processes. Our answer comprises a number of assets that form a comprehensive scheduling solution for distributed computing infrastructures. Based on a requirement analysis of application scenarios we provide a concept consisting of an automated scheduling process and the respective generic scheduling architecture supporting it. Process and architecture are based on four core models as there are a model to describe the activities to be executed, an information model to capture the capabilities of the infrastructure, a model to handle the life-cycle of service level agreements, which are the foundation for elaborated service management solutions, and a specific scheduling model capturing the specifics of state-of-the-art distributed systems. We deliver, in addition to concept and models, realisations of our solutions that demonstrate their applicability in different application scenarios spanning grid-like academic as well as financial service infrastructures. Last, but not least, we evaluate our scheduling model through simulations of artificial as well as realistic workload traces thus showing the feasibility of the approach and the implications of its usage. The work at hand therefore offers a blueprint for developers of scheduling solutions for state-of-the-art distributed computing infrastructures. It contributes essential building blocks to realise such solutions and provides an important step to integrate them into IT service management solutions

Co-Allocating Compute and Network Resources

Distributed applications or workflows need to access and use compute, storage and network resources simultaneously or chronologically coordinated respectively. Examples are distributed multi-physics simulations that use the combined computational performance and data storage of multiple clusters. A coordinated reservation and allocation of the resources is a prerequisite for the efficient use of such resources. This contribution describes the components of a system that provides Grid users with this functionality. The Grid middleware UNICORE is extended to access a MetaScheduling Service (MSS) performing orchestration of resources of different administrative domains, using advance reservation capability of local resource management systems (RMS) - including network connections for which ARGON serves as RMS. ARGON leverages Bandwidth on Demand, a cornerstone of next generation Grid enabled optical networks rendering the network to a first class Grid resource