Semantic Description, Publication and Discovery of Workflows in myGrid

The bioinformatics scientific process relies on in silico experiments, which are experiments executed in full in a computational environment. Scientists wish to encode the designs of these experiments as workflows because they provide minimal, declarative descriptions of the designs, overcoming many barriers to the sharing and re-use of these designs between scientists and enable the use of the most appropriate services available at any one time. We anticipate that the number of workflows will increase quickly as more scientists begin to make use of existing workflow construction tools to express their experiment designs. Discovery then becomes an increasingly hard problem, as it becomes more difficult for a scientist to identify the workflows relevant to their particular research goals amongst all those on offer. While many approaches exist for the publishing and discovery of services, there have been few attempts to address where and how authors of experimental designs should advertise the availability of their work or how relevant workflows can be discovered with minimal effort from the user. As the users designing and adapting experiments will not necessarily have a computer science background, we also have to consider how publishing and discovery can be achieved in such a way that they are not required to have detailed technical knowledge of workflow scripting languages. Furthermore, we believe they should be able to make use of others' expert knowledge (the semantics) of the given scientific domain. In this paper, we define the issues related to the semantic description, publishing and discovery of workflows, and demonstrate how the architecture created by the myGrid project aids scientists in this process. We give a walk-through of how users can construct, publish, annotate, discover and enact workflows via the user interfaces of the myGrid architecture; we then describe novel middleware protocols, making use of the Semantic Web technologies RDF and OWL to support workflow publishing and discovery

On the construction of decentralised service-oriented orchestration systems

Modern science relies on workflow technology to capture, process, and analyse data obtained from scientific instruments. Scientific workflows are precise descriptions of experiments in which multiple computational tasks are coordinated based on the dataflows between them. Orchestrating scientific workflows presents a significant research challenge: they are typically executed in a manner such that all data pass through a centralised computer server known as the engine, which causes unnecessary network traffic that leads to a performance bottleneck. These workflows are commonly composed of services that perform computation over geographically distributed resources, and involve the management of dataflows between them. Centralised orchestration is clearly not a scalable approach for coordinating services dispersed across distant geographical locations. This thesis presents a scalable decentralised service-oriented orchestration system that relies on a high-level data coordination language for the specification and execution of workflows. This system’s architecture consists of distributed engines, each of which is responsible for executing part of the overall workflow. It exploits parallelism in the workflow by decomposing it into smaller sub-workflows, and determines the most appropriate engines to execute them using computation placement analysis. This permits the workflow logic to be distributed closer to the services providing the data for execution, which reduces the overall data transfer in the workflow and improves its execution time. This thesis provides an evaluation of the presented system which concludes that decentralised orchestration provides scalability benefits over centralised orchestration, and improves the overall performance of executing a service-oriented workflow

Aplicación de la simulación en tiempo real para mejorar la calidad de servicio del middleware

La utilización de aplicaciones de diferente naturaleza dentro de un mismo entorno, entorno heterogéneo, se está extendiendo gracias a la incorporación de técnicas de virtualización a los servidores. Compartir un servidor ofrece ventajas sobretodo en términos de eficiencia de energía, utilización del espacio o mantenimiento. La virtualización añade ventajas en la separación de las diferentes aplicaciones o entornos. Aún así los gestores de recursos para entornos heterogéneos tienen como principal dificultad ofrecer calidad de servicio (QoS) a diferentes aplicaciones, entornos o cargas. Una aplicación que realice streaming y otra que realice cálculo intensivo, normalmente , no colisionaran ya que los recursos utilizados son diferentes. Por el otro lado, colisionaran dos aplicaciones que trabajen con la CPU.Nuestra propuesta ofrece la posibilidad de introducir dentro de estos gestores de recursos la capacidad de predecir este tipo de entornos, en concreto transaccionales y Grid, para aumentar la QoS y el rendimiento. Las predicciones han de utilizar técnicas de simulación ya que la mayoria de las veces el sistema no será representable mediante técnicas analíticas, por ser un sistema saturado o tener características difíciles de representar.La simulación es una técnica utilizada para predecir el comportamiento de sistemas en multitud de áreas. Las simulaciones de componentes hardware son muy comunes, dado el coste de construcción de los sistemas simulados (procesadores, memorias...). Sin embargo, el uso de la simulación en entornos complejos, como es el middleware, y su aplicación en gestores de recursos tiene un uso muy bajo. Nosotros proponemos simulaciones ligeras capaces de obtener resultados utilizables en estos entornos.Entre las aportaciones y contribuciones de la tesis tenemos: (i) utilización de métodos de simulación para incrementar el rendimiento y la calidad de servicio de estos sistemas. (ii) ampliación de un sistema de monitorización global para aplicaciones mixtas (JAVA y C) que nos ofrece la posibilidad de conseguir información de lo que ocurre en el middleware y de relacionarlo con el sistema. (iii) creación de un gestor de recursos capaz de repartir los recursos en un entorno heterogéneo utilizando la predicción para tener en cuenta diferentes parámetros de calidad de servicio.En la tesis se muestran los mecanismos de creación de los distintos simuladores, las herramientas de obtención de datos y monitorización, así como mecanismos autónomos que pueden alimentarse de la predicción para producir mejores resultados. Los resultados obtenidos, con gran impacto en la QoS en el gestor creado para Globus, demuestran que los métodos aplicados en esta tesis pueden ser válidos para crear gestores de recursos inteligentes, alimentados de las predicciones del sistema para tomar decisiones. Finalmente, utilizamos las simulaciones realizadas incorporándolas dentro de un prototipo de gestor de recursos heterogéneo capaz de repartir los recursos entre un entorno transaccional y un entorno Grid dentro del mismo servidor.Using different applications inside the same environment, heterogeneous environment, is getting more and more usual due the incorporation of the virtualization inside servers. Sharing a server offer advantages in different levels: energy, space, management. Virtualization helps to separate different applications or environments. On the other hand, resource managers have as principal issue offer Quality of Service for different applications, environments or workloads. A streaming server and a CPU intensive application would not collide; the resources they need are different. However, two applications that need CPU processing power will collide.Our proposal offers the possibility to introduce inside the resource manager the capacity to predict these environments. We will work with transactional and Grid environments, and we will increase the QoS and the performance. We need to use simulation techniques for our predictions because a large number of times the system won't be able to be modelled with analytic techniques, for being a saturated system or having features that are hard to reproduce.Simulation is a technique used to predict the behaviour of multiple systems in a large number of areas. Hardware simulations are very common because the building/testing cost of the simulated system (processor, memory, cache,...) is high. However, using simulation in complex environments, as the middleware, and its use in resource management is low. We propose light simulations that can obtain results that can be used in these environments.We will enumerate our contributions: (i) Use simulations to increase the performance and the QoS of those systems. (ii) Improve a global monitoring system for mixed applications (JAVA and C) that gives us information about what happens in the middleware and in the system. (iii) Build a resource manager that can share the resources in a heterogeneous environment an use the prediction to ensure the different QoS parameters that we provide.In the thesis we show how we built the different simulators, the different tools to obtain information and monitorize the applications, and finally the autonomic mechanisms that can feed with the prediction to obtain better results. Results obtained, with great success in the case of the resource manager created for Globus, show and demonstrate that the applied methods in this thesis are suitable to create intelligent resource managers, fed with predictions of the system to take decisions. Finally, we add the built simulations inside a heterogeneous resource manager that shares resources between a transactional environment and a Grid environment inside the same server

Development of a grid service for multi-objective design optimisation

The emerging grid technology is receiving great attention from researchers and applications that need computational and data capabilities to enhance performance and efficiency. Multi-Objective Design Optimisation (MODO) is computationally and data challenging. The challenges become even more with the emergence of evolutionary computing (EC) techniques which produce multiple solutions in a single simulation run. Other challenges are the complexity in mathematical models and multidisciplinary involvement of experts, thus making MODO collaborative and interactive in nature. These challenges call for a problem solving environment (P SE) that can provide computational and optimisation resources to MODO experts as services. Current PSEs provide only the technical specifications of the services which is used by programmers and do not have service specifications for designers that use the system to support design optimisation as services. There is need for PSEs to have service specification document that describes how the services are provided to the end users. Additionally, providing MODO resources as services enabled designers to share resources that they do not have through service subscription. The aim of this research is to develop specifications and architecture of a grid service for MODO. The specifications provide the service use cases that are used to build MODO services. A service specification document is proposed and this enables service providers to follow a process for providing services to end users. In this research, literature was reviewed and industry survey conducted. This was followed by the design, development, case study and validation. The research studied related PSEs in literature and industry to come up with a service specification document that captures the process for grid service definition. This specification was used to develop a framework for MODO applications. An architecture based on this framework was proposed and implemented as DECGrid (Decision Engineering Centre Grid) prototype. Three real-life case studies were used to validate the prototype. The results obtained compared favourably with the results in literature. Different scenarios for using the services among distributed design experts demonstrated the computational synergy and efficiency in collaboration. The mathematical model building service and optimisation service enabled designers to collaboratively build models using the collaboration service. This helps designers without optimisation knowledge to perform optimisation. The key contributions in this research are the service specifications that support MODO, the framework developed which provides the process for definining the services and the architecture used to implement the framework. The key limitations of the research are the use of only engineering design optimisation case studies and the prototype is not tested in industry.EThOS - Electronic Theses Online ServiceGBUnited Kingdo