Evaluation of Reallocation Heuristics for Moldable Tasks in Computational Dedicated and non Dedicated Grids

Grid services often consist of remote sequential or rigid parallel application executions. However, moldable parallel applications, linear algebra solvers for example, are of great interest but requires dynamic tuning which has mostly to be done interactively if performances are needed. Thus, their grid execution depends on a remote and transparent submission to a possibly different batch scheduler on each site, and means an automatic tuning of the job according to the local load. In this report we study the benefits of having a middleware able to automatically submit and reallocate requests from one site to another when it is also able to configure the services by tuning their number of processors and their walltime. In this context, we evaluate the benefits of such mechanisms on four multi-cluster Grid setups, where the platform is either composed of several heterogeneous or homogeneous, dedicated or non dedicated clusters. Different scenarios are explored using simulations of real cluster traces from different origins. Results show that a simple scheduling heuristic is good and often the best. Indeed, it is faster and thus can take more jobs into account while having a small execution time. Moreover, users can expect more jobs finishing sooner and a gain on the average job response time between 10\% and 40\% in most cases if this reallocation mechanism combined to auto-tuning capabilities is implemented in a Grid framework. The implementation and the maintenance of this heuristic coupled to the migration mechanism in a Grid middleware is also simpler because less transfers are involved.L'appel à des services présents sur les grilles de calcul correspondent généralement à l'exécution d'une application séquentielle ou rigide. Cependant, il est possible d'avoir des applications parallèles moldables, telles que des solveurs linéaires, qui sont d'un grand intérêt, mais qui demandent une adaptation dynamique pour obtenir de bonnes performances. Leur exécution nécessite donc d'avoir un accès distant et transparent à différents gestionnaires de ressources, demandant donc une adaptation automatique de l'application en fonction de la charge locale. Dans ce rapport, nous étudions les bénéfices découlant de l'utilisation d'un intergiciel de grille capable de soumettre et de réallouer des requêtes d'un site à l'autre tout en configurant automatiquement les services en choisissant le nombre de processeurs ainsi que la durée d'exécution estimée. Dans ce contexte, nous évaluons les gains apportés par de tels mécanismes sur quatre grilles de calcul différentes où la plate-forme est composée de plusieurs grappes, homogène ou hétérogènes, dédiées ou non. Nous explorons différents scénarios par la simulation de traces de tâches provenant de réelles exécutions. Les résultats montrent que l'utilisation d'une heuristique d'ordonnancement simple est efficace, souvent amplement suffisante, voire la meilleure. En effet, elle est plus rapide à l'exécution et permet de prendre plus de requêtes en compte. Les utilisateurs peuvent espérer une majorité de requêtes terminant plus tôt si elle est utilisée, ainsi qu'une réduction du temps d'attente du résultat d'entre 10\% et 40\% dans la plupart des cas lorsque le mécanisme de réallocation couplé à l'adaptation automatique sont présents dans l'intergiciel. De plus, l'implantation et la maintenance de cette heuristique couplée au mécanisme de migration de tâches dans un intergiciel de grille est aussi plus facile car moins de tranferts sont nécessaires

Evaluation of the OGF GridRPC Data Management library, and study of its integration into an International Sparse Linear Algebra Expert System

International audienceThe Data Management API for the GridRPC describes an optional API that extends the GridRPC standard. It provides a minimal subset of functions to handle a large set of data operations, among which movement, replication, migration and stickyness. We already showed that its use leads to 1) reduced time to completion of application, since useless transfers are avoided; 2) improved feasibility of some computations, depending on the availability of services and/or storage space constraints; 3) complete code portability between two GridRPC middleware; and 4) seamless interoperability, in our example between the French GridRPC middleware DIET and the Japanese middleware Ninf, distributed on French and Japanese administrative domains respectively, leading to both of them contributing to the same calculus, their respective servers sharing only data through our implementation of the GridRPC DM API. We have extended the implementation of the library and a further integration has been made available into DIET as a back-end of its data manager Dagda. We thus present how the library is used in the International Sparse Linear Algebra Expert System GridTLSE which manages entire expertises for the user, including data transfers, tasks executions, and graphical charts, to help analysing the overall execution. GridTLSE relies on DIET to distribute computations and thus can benefit from the persistency functionalities to provide scientists with faster results when their expertises require the same input matrices. In addition, with the possibility for two middleware to interact in a seamless way as long as they’re using an implementation of the GridRPC Data Management API, new architecture of different domains can easily be integrated to the expert system and thus helps the linear algebra community

Study of the behaviour of heuristics relying on the Historical Trace Manager in a (multi)client-agent-server system

We compare some dynamic scheduling heuristics that have shown good performances on simulation study against MCT on experiments on real solving platforms. The heuristics rely on a prediction module, the Historical Trace Manager. They have been implemented in NetSolve, a Problem Solver Environment built on the client-agent-server model. Numerous different scenarios have been examined and many metrics have been considered. We show that the predicting module allows a better precision in task duration estimation and that our heuristics optimize several metrics at the same time while outperforming MCT

New Dynamic Heuristics in the Client-Agent-Server Model

Colloque avec actes et comité de lecture. internationale.International audienceMCT is a widely used heuristic for scheduling tasks onto grid platforms. However, when dealing with many tasks, MCT tends to dramatically delay already mapped task completion time, while scheduling a new task. In this paper we propose heuristics based on two features: the historical trace manager that simulates the environment and the perturbation that defines the impact a new allocated task has on already mapped tasks. Our simulations and experiments on a real environment show that the proposed heuristics outperform MCT

Parallel constraint-based local search on the HA8000 supercomputer (abstract)

We present a parallel implementation of a constraint-based local search algorithm and investigate its performance re- sults on hardware with several hundreds of processors

Scheduling on the Grid : Historical Trace and Dynamic Heuristics

We present a historical trace manager and new dynamic scheduling heuristics that can be used, and are studied, in the client-agent-server model on the `grid'. These heuristics rely on the common acknowledgment of the characteristics of the tasks submitted to the agent, but also on the construction of the underlying historical trace of the different tasks submitted to each server. We study each heuristic and compare them on several metrics to an instantiation of MCT (Minimum Completion time), chosen as reference heuristic. The simulation experiments we have conducted show that they are likely to give good results when tested in a real environment

Cosmological Simulations using Grid Middleware

One way to access the aggregated power of a collection of heterogeneous machines is to use a grid middleware, such as DIET, GridSolve or NINF. It addresses the problem of monitoring the resources, of handling the submissions of jobs and as an example the inherent transfer of input and output data, in place of the user. In this paper we present how to run cosmological simulations using the RAMSES application along with the DIET middleware. We will describe how to write the corresponding DIET client and server. The remainder of the paper is organized as follows: Section 2 presents the DIET middleware. Section 3 describes the RAMSES cosmological software and simulations, and how to interface it with DIET. We show how to write a client and a server in Section 4. Finally, Section 5 presents the experiments realized on Grid'5000, the French Research Grid, and we conclude in Section 6.Comment: submitted Nov 200

Simbatch: an API for simulating and predicting the performance of parallel resources and batch systems

The study of scheduling algorithms for parallel tasks in a grid computing context either neglects local reservation systems which manage parallel resources, either suppose that they use a First Come First Served strategy, or the experimental model does not handle parallel tasks. In this report, we describe an API built in the grid simulation tool Simgrid. It offers core functionalities to simulate in a realistic way parallel resources and batch reservation systems. Simbatch simulation experiments show an error rate inferior to 1% compared to real life experiments conducted with the OAR batch manager

Parallel and Distributed Stream Processing: Systems Classification and Specific Issues

Deploying an infrastructure to execute queries on distributed data streams sources requires to identify a scalable and robust solution able to provide results which can be qualified. Last decade, different Data Stream Management Systems have been designed by exploiting new paradigm and technologies to improve performances of solutions facing specific features of data streams and their growing number. However, some tradeoffs are often achieved between performance of the processing, resources consumption and quality of results. This survey 5 suggests an overview of existing solutions among distributed and parallel systems classified according to criteria able to allow readers to efficiently identify relevant existing Distributed Stream Management Systems according to their needs ans resources

Large-scale parallelism for constraint-based local search: the costas array case study

International audienceWe present the parallel implementation of a constraint-based Local Search algorithm and investigate its performance on several hardware plat-forms with several hundreds or thousands of cores. We chose as the basis for these experiments the Adaptive Search method, an efficient sequential Local Search method for Constraint Satisfaction Problems (CSP). After preliminary experiments on some CSPLib benchmarks, we detail the modeling and solving of a hard combinatorial problem related to radar and sonar applications: the Costas Array Problem. Performance evaluation on some classical CSP bench-marks shows that speedups are very good for a few tens of cores, and good up to a few hundreds of cores. However for a hard combinatorial search problem such as the Costas Array Problem, performance evaluation of the sequential version shows results outperforming previous Local Search implementations, while the parallel version shows nearly linear speedups up to 8,192 cores. The proposed parallel scheme is simple and based on independent multi-walks with no communication between processes during search. We also investigated a cooperative multi-walk scheme where processes share simple information, but this scheme does not seem to improve performance