The OMII Software Distribution

This paper describes the work carried out at the Open Middleware Infrastructure Institute (OMII) and the key elements of the OMII software distribution that have been developed in collaboration with members of the Managed Programme Initiative. The main objective of the OMII is to preserve and consolidate the achievements of the UK e-Science Programme by collecting, maintaining and improving the software modules that form the key components of a generic Grid middleware. Recently, the activity at Southampton has been extended beyond 2009 through a new project, OMII-UK, that forms a partnership that now includes the OGSA-DAI activities at Edinburgh and the myGrid project at Manchester

Integration of chargeable web services into engineering applications

We discuss an engineering case-study in which we address the issues of integration and secure consumption of a specialist software package for finite element meshing through a web service based interface. This work focuses on demonstrating the end-to-end requirements of a framework allowing the transparent integration of the web service into the design workflow of an engineering user. After detailing an example engineering problem, we discuss server-side issues and required modules such as local accounting, security and resource management which provide a sample framework for a chargeable web service. We then demonstrate how the client can easily interface to and consume the web service from within the engineering scripting language Matlab

Probabilistic grid scheduling based on job statistics and monitoring information

This transfer thesis presents a novel, probabilistic approach to scheduling applications on computational Grids based on their historical behaviour, current state of the Grid and predictions of the future execution times and resource utilisation of such applications. The work lays a foundation for enabling a more intuitive, user-friendly and effective scheduling technique termed deadline scheduling. Initial work has established motivation and requirements for a more efficient Grid scheduler, able to adaptively handle dynamic nature of the Grid resources and submitted workload. Preliminary scheduler research identified the need for a detailed monitoring of Grid resources on the process level, and for a tool to simulate non-deterministic behaviour and statistical properties of Grid applications. A simulation tool, GridLoader, has been developed to enable modelling of application loads similar to a number of typical Grid applications. GridLoader is able to simulate CPU utilisation, memory allocation and network transfers according to limits set through command line parameters or a configuration file. Its specific strength is in achieving set resource utilisation targets in a probabilistic manner, thus creating a dynamic environment, suitable for testing the scheduler’s adaptability and its prediction algorithm. To enable highly granular monitoring of Grid applications, a monitoring framework based on the Ganglia Toolkit was developed and tested. The suite is able to collect resource usage information of individual Grid applications, integrate it into standard XML based information flow, provide visualisation through a Web portal, and export data into a format suitable for off-line analysis. The thesis also presents initial investigation of the utilisation of University College London Central Computing Cluster facility running Sun Grid Engine middleware. Feasibility of basic prediction concepts based on the historical information and process meta-data have been successfully established and possible scheduling improvements using such predictions identified. The thesis is structured as follows: Section 1 introduces Grid computing and its major concepts; Section 2 presents open research issues and specific focus of the author’s research; Section 3 gives a survey of the related literature, schedulers, monitoring tools and simulation packages; Section 4 presents the platform for author’s work – the Self-Organising Grid Resource management project; Sections 5 and 6 give detailed accounts of the monitoring framework and simulation tool developed; Section 7 presents the initial data analysis while Section 8.4 concludes the thesis with appendices and references

A Semantic Web Based Approach to Knowledge Management for Grid Applications

Knowledge has become increasingly important to support intelligent process automation and collaborative problem solving in large-scale science over the Internet. This paper addresses distributed knowledge management, its approach and methodology, in the context of grid application. We start by analyzing the nature of grid computing and its requirements for knowledge support; then, we discuss knowledge characteristics and the challenges for knowledge management on the grid. A semantic Web-based approach is proposed to tackle the six challenges of the knowledge lifecycle - namely, those of acquiring, modeling, retrieving, reusing, publishing, and maintaining knowledge. To facilitate the application of the approach, a systematic methodology is conceived and designed to provide a general implementation guideline. We use a real-world Grid application, the GEODISE project, as a case study in which the core semantic Web technologies such as ontologies, semantic enrichment, and semantic reasoning are used for knowledge engineering and management. The case study has been fully implemented and deployed through which the evaluation and validation for the approach and methodology have been performe

Development of a grid service for multi-objective design optimisation

The emerging grid technology is receiving great attention from researchers and applications that need computational and data capabilities to enhance performance and efficiency. Multi-Objective Design Optimisation (MODO) is computationally and data challenging. The challenges become even more with the emergence of evolutionary computing (EC) techniques which produce multiple solutions in a single simulation run. Other challenges are the complexity in mathematical models and multidisciplinary involvement of experts, thus making MODO collaborative and interactive in nature. These challenges call for a problem solving environment (P SE) that can provide computational and optimisation resources to MODO experts as services. Current PSEs provide only the technical specifications of the services which is used by programmers and do not have service specifications for designers that use the system to support design optimisation as services. There is need for PSEs to have service specification document that describes how the services are provided to the end users. Additionally, providing MODO resources as services enabled designers to share resources that they do not have through service subscription. The aim of this research is to develop specifications and architecture of a grid service for MODO. The specifications provide the service use cases that are used to build MODO services. A service specification document is proposed and this enables service providers to follow a process for providing services to end users. In this research, literature was reviewed and industry survey conducted. This was followed by the design, development, case study and validation. The research studied related PSEs in literature and industry to come up with a service specification document that captures the process for grid service definition. This specification was used to develop a framework for MODO applications. An architecture based on this framework was proposed and implemented as DECGrid (Decision Engineering Centre Grid) prototype. Three real-life case studies were used to validate the prototype. The results obtained compared favourably with the results in literature. Different scenarios for using the services among distributed design experts demonstrated the computational synergy and efficiency in collaboration. The mathematical model building service and optimisation service enabled designers to collaboratively build models using the collaboration service. This helps designers without optimisation knowledge to perform optimisation. The key contributions in this research are the service specifications that support MODO, the framework developed which provides the process for definining the services and the architecture used to implement the framework. The key limitations of the research are the use of only engineering design optimisation case studies and the prototype is not tested in industry.EThOS - Electronic Theses Online ServiceGBUnited Kingdo

Autonomous grid scheduling using probabilistic job runtime scheduling

Computational Grids are evolving into a global, service-oriented architecture – a universal platform for delivering future computational services to a range of applications of varying complexity and resource requirements. The thesis focuses on developing a new scheduling model for general-purpose, utility clusters based on the concept of user requested job completion deadlines. In such a system, a user would be able to request each job to finish by a certain deadline, and possibly to a certain monetary cost. Implementing deadline scheduling is dependent on the ability to predict the execution time of each queued job, and on an adaptive scheduling algorithm able to use those predictions to maximise deadline adherence. The thesis proposes novel solutions to these two problems and documents their implementation in a largely autonomous and self-managing way. The starting point of the work is an extensive analysis of a representative Grid workload revealing consistent workflow patterns, usage cycles and correlations between the execution times of jobs and its properties commonly collected by the Grid middleware for accounting purposes. An automated approach is proposed to identify these dependencies and use them to partition the highly variable workload into subsets of more consistent and predictable behaviour. A range of time-series forecasting models, applied in this context for the first time, were used to model the job execution times as a function of their historical behaviour and associated properties. Based on the resulting predictions of job runtimes a novel scheduling algorithm is able to estimate the latest job start time necessary to meet the requested deadline and sort the queue accordingly to minimise the amount of deadline overrun. The testing of the proposed approach was done using the actual job trace collected from a production Grid facility. The best performing execution time predictor (the auto-regressive moving average method) coupled to workload partitioning based on three simultaneous job properties returned the median absolute percentage error centroid of only 4.75%. This level of prediction accuracy enabled the proposed deadline scheduling method to reduce the average deadline overrun time ten-fold compared to the benchmark batch scheduler. Overall, the thesis demonstrates that deadline scheduling of computational jobs on the Grid is achievable using statistical forecasting of job execution times based on historical information. The proposed approach is easily implementable, substantially self-managing and better matched to the human workflow making it well suited for implementation in the utility Grids of the future

Stochastic scheduling and workload allocation : QoS support and profitable brokering in computing grids

Abstract: The Grid can be seen as a collection of services each of which performs some functionality. Users of the Grid seek to use combinations of these services to perform the overall task they need to achieve. In general this can be seen as aset of services with a workflow document describing how these services should be combined. The user may also have certain constraints on the workflow operations, such as execution time or cost ----t~ th~ user, specified in the form of a Quality of Service (QoS) document. The users . submit their workflow to a brokering service along with the QoS document. The brokering service's task is to map any given workflow to a subset of the Grid services taking the QoS and state of the Grid into account -- service availability and performance. We propose an approach for generating constraint equations describing the workflow, the QoS requirements and the state of the Grid. This set of equations may be solved using Mixed-Integer Linear Programming (MILP), which is the traditional method. We further develop a novel 2-stage stochastic MILP which is capable of dealing with the volatile nature of the Grid and adapting the selection of the services during the lifetime of the workflow. We present experimental results comparing our approaches, showing that the . 2-stage stochastic programming approach performs consistently better than other traditional approaches. Next we addresses workload allocation techniques for Grid workflows in a multi-cluster Grid We model individual clusters as MIMIk. queues and obtain a numerical solutio~ for missed deadlines (failures) of tasks of Grid workflows. We also present an efficient algorithm for obtaining workload allocations of clusters. Next we model individual cluster resources as G/G/l queues and solve an optimisation problem that minimises QoS requirement violation, provides QoS guarantee and outperforms reservation based scheduling algorithms. Both approaches are evaluated through an experimental simulation and the results confirm that the proposed workload allocation strategies combined with traditional scheduling algorithms performs considerably better in terms of satisfying QoS requirements of Grid workflows than scheduling algorithms that don't employ such workload allocation techniques. Next we develop a novel method for Grid brokers that aims at maximising profit whilst satisfying end-user needs with a sufficient guarantee in a volatile utility Grid. We develop a develop a 2-stage stochastic MILP which is capable of dealing with the volatile nature . of the Grid and obtaining cost bounds that ensure that end-user cost is minimised or satisfied and broker's profit is maximised with sufficient guarantee. These bounds help brokers know beforehand whether the budget limits of end-users can be satisfied and. if not then???????? obtain appropriate future leases from service providers. Experimental results confirm the efficacy of our approach.Imperial Users onl

A model predictive control-based energy management scheme for hybrid storage system in islanded microgrids

Model predictive control (MPC) facilitates online optimal resource scheduling in electrical networks, thermal systems, water networks, process industry to name a few. In electrical systems, the capability of MPC can be used not only to minimise operating costs but also to improve renewable energy utilisation and energy storage system degradation. This work assesses the application of MPC for energy management in an islanded microgrid with PV generation and hybrid storage system composed of battery, supercapacitor and regenerative fuel cell. The objective is to improve the utilisation of renewable generation, the operational efficiency of the microgrid and the reduction in rate of degradation of storage systems. The improvements in energy scheduling, achieved with MPC, are highlighted through comparison with a heuristic based method, like Fuzzy inference. Simulated behaviour of an islanded microgrid with the MPC and fuzzy based energy management schemes will be studied for the same. Apart from this, the study also carries out an analysis of the computational demand resulting from the use of MPC in the energy management stage. It is concluded that, compared to heuristic methods, MPC ensures improved performance in an islanded microgrid.This work was supported in part by the European Union’s Horizon 2020 Research and Innovation Program under the Marie Skłodowska Curie under Grant 675318 (INCITE), in part by the Spanish State Research Agency through the Maria de Maeztu Seal of Excellence to IRI under Grant MDM-2016-0656, and in part by the Spanish National Project DOVELAR (MCIU/AEI/FEDER, UE) under Grant RTI2018-096001-B-C32.Peer ReviewedPostprint (published version