Hybrid Meta-heuristic Algorithms for Static and Dynamic Job Scheduling in Grid Computing

The term ’grid computing’ is used to describe an infrastructure that connects geographically distributed computers and heterogeneous platforms owned by multiple organizations allowing their computational power, storage capabilities and other resources to be selected and shared. Allocating jobs to computational grid resources in an efficient manner is one of the main challenges facing any grid computing system; this allocation is called job scheduling in grid computing. This thesis studies the application of hybrid meta-heuristics to the job scheduling problem in grid computing, which is recognized as being one of the most important and challenging issues in grid computing environments. Similar to job scheduling in traditional computing systems, this allocation is known to be an NPhard problem. Meta-heuristic approaches such as the Genetic Algorithm (GA), Variable Neighbourhood Search (VNS) and Ant Colony Optimisation (ACO) have all proven their effectiveness in solving different scheduling problems. However, hybridising two or more meta-heuristics shows better performance than applying a stand-alone approach. The new high level meta-heuristic will inherit the best features of the hybridised algorithms, increasing the chances of skipping away from local minima, and hence enhancing the overall performance. In this thesis, the application of VNS for the job scheduling problem in grid computing is introduced. Four new neighbourhood structures, together with a modified local search, are proposed. The proposed VNS is hybridised using two meta-heuristic methods, namely GA and ACO, in loosely and strongly coupled fashions, yielding four new sequential hybrid meta-heuristic algorithms for the problem of static and dynamic single-objective independent batch job scheduling in grid computing. For the static version of the problem, several experiments were carried out to analyse the performance of the proposed schedulers in terms of minimising the makespan using well known benchmarks. The experiments show that the proposed schedulers achieved impressive results compared to other traditional, heuristic and meta-heuristic approaches selected from the bibliography. To model the dynamic version of the problem, a simple simulator, which uses the rescheduling technique, is designed and new problem instances are generated, by using a well-known methodology, to evaluate the performance of the proposed hybrid schedulers. The experimental results show that the use of rescheduling provides significant improvements in terms of the makespan compared to other non-rescheduling approaches

Managing Uncertainty: A Case for Probabilistic Grid Scheduling

The Grid technology is evolving into a global, service-orientated architecture, a universal platform for delivering future high demand computational services. Strong adoption of the Grid and the utility computing concept is leading to an increasing number of Grid installations running a wide range of applications of different size and complexity. In this paper we address the problem of elivering deadline/economy based scheduling in a heterogeneous application environment using statistical properties of job historical executions and its associated meta-data. This approach is motivated by a study of six-month computational load generated by Grid applications in a multi-purpose Grid cluster serving a community of twenty e-Science projects. The observed job statistics, resource utilisation and user behaviour is discussed in the context of management approaches and models most suitable for supporting a probabilistic and autonomous scheduling architecture

A Multi-Criteria Meta-Fuzzy-Scheduler for Independent Tasks in Grid Computing

The paradigm of distributed computation in heterogeneous resources, grid computing, has given rise to a large amount of research on resource scheduling. This paper presents a Meta-Scheduler for grid computing that does not need any given information about tasks length or tasks arrival time unlike traditional dynamic heuristics. Our Meta-Scheduler is of multi-criteria type, because it solves two conflicting objectives: minimize the makespan of a set of tasks and distribute these tasks in a balanced way among the resources of the Grid. Experimental results using fuzzy scheduler show that, through our proposal, we achieve these two objectives and improve dynamic heuristics presented in prior literature

The LCG POOL Project, General Overview and Project Structure

The POOL project has been created to implement a common persistency framework for the LHC Computing Grid (LCG) application area. POOL is tasked to store experiment data and meta data in the multi Petabyte area in a distributed and grid enabled way. First production use of new framework is expected for summer 2003. The project follows a hybrid approach combining C++ Object streaming technology such as ROOT I/O for the bulk data with a transactionally safe relational database (RDBMS) store such as MySQL. POOL is based a strict component approach - as laid down in the LCG persistency and blue print RTAG documents - providing navigational access to distributed data without exposing details of the particular storage technology. This contribution describes the project breakdown into work packages, the high level interaction between the main pool components and summarizes current status and plans.Comment: Talk from the 2003 Computing in High Energy and Nuclear Physics (CHEP03), La Jolla, Ca, USA, March 2003, 5 pages. PSN MOKT00

DIANA Scheduling Hierarchies for Optimizing Bulk Job Scheduling

The use of meta-schedulers for resource management in large-scale distributed systems often leads to a hierarchy of schedulers. In this paper, we discuss why existing meta-scheduling hierarchies are sometimes not sufficient for Grid systems due to their inability to re-organise jobs already scheduled locally. Such a job re-organisation is required to adapt to evolving loads which are common in heavily used Grid infrastructures. We propose a peer-to-peer scheduling model and evaluate it using case studies and mathematical modelling. We detail the DIANA (Data Intensive and Network Aware) scheduling algorithm and its queue management system for coping with the load distribution and for supporting bulk job scheduling. We demonstrate that such a system is beneficial for dynamic, distributed and self-organizing resource management and can assist in optimizing load or job distribution in complex Grid infrastructures.Comment: 8 pages, 9 figures. Presented at the 2nd IEEE Int Conference on eScience & Grid Computing. Amsterdam Netherlands, December 200

Enhancing the genetic-based scheduling in computational grids by a structured hierarchical population

Independent Job Scheduling is one of the most useful versions of scheduling in grid systems. It aims at computing efficient and optimal mapping of jobs and/or applications submitted by independent users to the grid resources. Besides traditional restrictions, mapping of jobs to resources should be computed under high degree of heterogeneity of resources, the large scale and the dynamics of the system. Because of the complexity of the problem, the heuristic and meta-heuristic approaches are the most feasible methods of scheduling in grids due to their ability to deliver high quality solutions in reasonable computing time. One class of such meta-heuristics is Hierarchic Genetic Strategy (HGS). It is defined as a variant of Genetic Algorithms (GAs) which differs from the other genetic methods by its capability of concurrent search of the solution space. In this work, we present an implementation of HGS for Independent Job Scheduling in dynamic grid environments. We consider the bi-objective version of the problem in which makespan and flowtime are simultaneously optimized. Based on our previous work, we improve the HGS scheduling strategy by enhancing its main branching operations. The resulting HGS-based scheduler is evaluated under the heterogeneity, the large scale and dynamics conditions using a grid simulator. The experimental study showed that the HGS implementation outperforms existing GA-based schedulers proposed in the literature.Peer ReviewedPostprint (author's final draft

Meta-heuristically seeded genetic algorithm for independent job scheduling in grid computing

Grid computing is an infrastructure which connects geographically distributed computers owned by various organizations allowing their resources, such as computational power and storage capabilities, to be shared, selected, and aggregated. Job scheduling problem is one of the most difficult tasks in grid computing systems. To solve this problem efficiently, new methods are required. In this paper, a seeded genetic algorithm is proposed which uses a meta-heuristic algorithm to generate its initial population. To evaluate the performance of the proposed method in terms of minimizing the makespan, the Expected Time to Compute (ETC) simulation model is used to carry out a number of experiments. The results show that the proposed algorithm performs better than other selected techniques

Towards ensuring scalability, interoperability and efficient access control in a multi-domain grid-based environment

The application of grid computing has been hampered by three basic challenges: scalability, interoperability and efficient access control which need to be optimized before a full-scale adoption of grid computing can take place. To address these challenges, a novel architectural model was designed for a multi-domain grid based environment (built on three domains). It was modelled using the dynamic role-based access control. The architecture’s framework assumes that each domain has an independent local security monitoring unit and a central security monitoring unit that monitors security for the entire grid. The architecture was evaluated using the Grid Security Services Simulator, a meta-query language and Java Runtime Environment 1.7.0.5 for implementing the workflows that define the model’s task. In terms of scalability, the results show that as the number of grid nodes increases, the average turnaround time reduces, and thereby increases the number of service requesters (grid users) on the grid. Grid middleware integration across various domains as well as the appropriate handling of authentication and authorisation through a local security monitoring unit and a central security monitoring unit proved that the architecture is interoperable. Finally, a case study scenario used for access control across the domains shows the efficiency of the role based access control approach used for achieving appropriate access to resources. Based on the results obtained, the proposed framework has proved to be interoperable, scalable and efficiently suitable for enforcing access control within the parameters evaluated.Department of HE and Training approved lis

A game-theoretic and hybrid genetic meta-heuristics model for security-assured scheduling of independent jobs in computational grids

Scheduling independent tasks in Computational Grids commonly arises in many Grid-enabled large scale applications. Much of current research in this domain is focused on the improvement of the efficiency of the Grid schedulers, both at global and local levels, which is the basis for Grid systems to leverage large computing capacities. However, unlike traditional scheduling, in Grid systems security requirements are very important to scheduling tasks/applications to Grid resources. The objective is thus to achieve efficient and secure allocation of tasks to machines. In this paper we propose a new model for secure scheduling at the Grid sites by combining game-theoretic and genetic-based meta-heuristic approaches. The game-theoretic model takes into account the realistic feature that Grid users usually perform independently of each other. The scheduling problem is then formalized as a noncooperative non-zero sum game with Nash equilibria as the solutions. The game cost function is minimized, at global and user levels, by using four genetic-based hybrid meta-heuristics. We have evaluated the proposed model through a static benchmark of instances, for which we have measured two basic metrics, namely the makespan and flowtime. The obtained results suggest that it is more resilient for the Grid users (and local schedulers) to tolerate some job delays defined as additional scheduling cost due to security requirements instead of taking a risk of allocating at unreliable resources.Peer ReviewedPostprint (published version

Designing Computational Grids Using Best Practices in Software Architecture

The basic principle of sharing and collaborative work by geographically separated computers is known by several names such as meta computing, scalable computing, cluster computing, internet computing and this has today metamorphosed into a new term known as grid computing. Grid computing is proving to be promising method of HPC, which is packaged with many challenges. This paper elucidates the role that pattern can play in architecting complex systems with specific reference to grid computing. We provide descriptions of a set of well-engineered patterns that the practicing developer can apply to crafting his or her own specific applications. We develop the Software Requirements Specification (SRS), with an attempt to drive to effectual design specifications for use by any grid developer. We analyze the grid using an Object Oriented approach and present the design using the unified Modeling Language (UML) which itself helps the identification of patterns at different phases