Evaluation of Reallocation Heuristics for Moldable Tasks in Computational Dedicated and non Dedicated Grids

Grid services often consist of remote sequential or rigid parallel application executions. However, moldable parallel applications, linear algebra solvers for example, are of great interest but requires dynamic tuning which has mostly to be done interactively if performances are needed. Thus, their grid execution depends on a remote and transparent submission to a possibly different batch scheduler on each site, and means an automatic tuning of the job according to the local load. In this report we study the benefits of having a middleware able to automatically submit and reallocate requests from one site to another when it is also able to configure the services by tuning their number of processors and their walltime. In this context, we evaluate the benefits of such mechanisms on four multi-cluster Grid setups, where the platform is either composed of several heterogeneous or homogeneous, dedicated or non dedicated clusters. Different scenarios are explored using simulations of real cluster traces from different origins. Results show that a simple scheduling heuristic is good and often the best. Indeed, it is faster and thus can take more jobs into account while having a small execution time. Moreover, users can expect more jobs finishing sooner and a gain on the average job response time between 10\% and 40\% in most cases if this reallocation mechanism combined to auto-tuning capabilities is implemented in a Grid framework. The implementation and the maintenance of this heuristic coupled to the migration mechanism in a Grid middleware is also simpler because less transfers are involved.L'appel à des services présents sur les grilles de calcul correspondent généralement à l'exécution d'une application séquentielle ou rigide. Cependant, il est possible d'avoir des applications parallèles moldables, telles que des solveurs linéaires, qui sont d'un grand intérêt, mais qui demandent une adaptation dynamique pour obtenir de bonnes performances. Leur exécution nécessite donc d'avoir un accès distant et transparent à différents gestionnaires de ressources, demandant donc une adaptation automatique de l'application en fonction de la charge locale. Dans ce rapport, nous étudions les bénéfices découlant de l'utilisation d'un intergiciel de grille capable de soumettre et de réallouer des requêtes d'un site à l'autre tout en configurant automatiquement les services en choisissant le nombre de processeurs ainsi que la durée d'exécution estimée. Dans ce contexte, nous évaluons les gains apportés par de tels mécanismes sur quatre grilles de calcul différentes où la plate-forme est composée de plusieurs grappes, homogène ou hétérogènes, dédiées ou non. Nous explorons différents scénarios par la simulation de traces de tâches provenant de réelles exécutions. Les résultats montrent que l'utilisation d'une heuristique d'ordonnancement simple est efficace, souvent amplement suffisante, voire la meilleure. En effet, elle est plus rapide à l'exécution et permet de prendre plus de requêtes en compte. Les utilisateurs peuvent espérer une majorité de requêtes terminant plus tôt si elle est utilisée, ainsi qu'une réduction du temps d'attente du résultat d'entre 10\% et 40\% dans la plupart des cas lorsque le mécanisme de réallocation couplé à l'adaptation automatique sont présents dans l'intergiciel. De plus, l'implantation et la maintenance de cette heuristique couplée au mécanisme de migration de tâches dans un intergiciel de grille est aussi plus facile car moins de tranferts sont nécessaires

Cosmological Simulations using Grid Middleware

One way to access the aggregated power of a collection of heterogeneous machines is to use a grid middleware, such as DIET, GridSolve or NINF. It addresses the problem of monitoring the resources, of handling the submissions of jobs and as an example the inherent transfer of input and output data, in place of the user. In this paper we present how to run cosmological simulations using the RAMSES application along with the DIET middleware. We will describe how to write the corresponding DIET client and server. The remainder of the paper is organized as follows: Section 2 presents the DIET middleware. Section 3 describes the RAMSES cosmological software and simulations, and how to interface it with DIET. We show how to write a client and a server in Section 4. Finally, Section 5 presents the experiments realized on Grid'5000, the French Research Grid, and we conclude in Section 6.Comment: submitted Nov 200

Evaluation of the OGF GridRPC Data Management library, and study of its integration into an International Sparse Linear Algebra Expert System

International audienceThe Data Management API for the GridRPC describes an optional API that extends the GridRPC standard. It provides a minimal subset of functions to handle a large set of data operations, among which movement, replication, migration and stickyness. We already showed that its use leads to 1) reduced time to completion of application, since useless transfers are avoided; 2) improved feasibility of some computations, depending on the availability of services and/or storage space constraints; 3) complete code portability between two GridRPC middleware; and 4) seamless interoperability, in our example between the French GridRPC middleware DIET and the Japanese middleware Ninf, distributed on French and Japanese administrative domains respectively, leading to both of them contributing to the same calculus, their respective servers sharing only data through our implementation of the GridRPC DM API. We have extended the implementation of the library and a further integration has been made available into DIET as a back-end of its data manager Dagda. We thus present how the library is used in the International Sparse Linear Algebra Expert System GridTLSE which manages entire expertises for the user, including data transfers, tasks executions, and graphical charts, to help analysing the overall execution. GridTLSE relies on DIET to distribute computations and thus can benefit from the persistency functionalities to provide scientists with faster results when their expertises require the same input matrices. In addition, with the possibility for two middleware to interact in a seamless way as long as they’re using an implementation of the GridRPC Data Management API, new architecture of different domains can easily be integrated to the expert system and thus helps the linear algebra community

Improvements and Study of the Accuracy of the Tasks Duration Predictor, New Heuristics

The Historical Trace Manager is a task duration predictor module embedded in the agent of a Problem Solving Environment relying on the client-agent-server. The HTM is introduced in and . In this paper, we explain some improvements built into the HTM and NetSolve, the Problem Soving Environment we use for our tests, in order to synchronize the HTM to the reality.We also introduce two new scheduling heuritics relying on the HTM information: Advanced HMCT and Minimum Length.We study the scheduling of several scenarios, including the simultaneous submissions of DAGS and independent tasks, on a real heterogeneous platform.The excellent behavior of the HTM validates its estimations of the duration of each task concurrently running in the system. It can consequently predict the contention tasks may have on each other if scheduled and executed concurrently on the same computing resource.Heuristics performances show the relevancy of the HTM information through the experiments: their ability of behaving with a constant quality between two executions of the same experiment as well as the quality of their respective scheduling choices to optimize several criteria at the same time. We also show that heuristics which rely on minimizing the contention give generally the best results regardless the criterion.We finally compare the behavior of the heuristics previously tested in to the one observed here with more precise information on the global system state due to the synchronization mechanisms. Surprisingly, in the time-shared model, it does not necessarily improve the job repartition among the servers, performances can consequently decrease and the utilization of the fastest servers can become critical

Parallel constraint-based local search on the HA8000 supercomputer (abstract)

We present a parallel implementation of a constraint-based local search algorithm and investigate its performance re- sults on hardware with several hundreds of processors

Analyse de sensibilité globale pour des modèles à paramètres dépendants

En ingénierie mécanique, la performance d'un système M est en général liée à la variabilité de sa réponse mécanique, Y=M(X) où X désigne les paramètres du modèle représentant le comportement du système. L'analyse de sensibilité permet d'identifier quels sont les paramètres Xi principalement responsable de la variabilité de Y. Pour des processus complexes, dont les paramètres sont dépendants, on propose de traiter le problème par une méthode basée sur l'analyse de la distribution probabiliste du résultat notée f(y). La méthode est appliquée à un exemple académique en thermomécanique

Study of the behaviour of heuristics relying on the Historical Trace Manager in a (multi)client-agent-server system

We compare some dynamic scheduling heuristics that have shown good performances on simulation study against MCT on experiments on real solving platforms. The heuristics rely on a prediction module, the Historical Trace Manager. They have been implemented in NetSolve, a Problem Solver Environment built on the client-agent-server model. Numerous different scenarios have been examined and many metrics have been considered. We show that the predicting module allows a better precision in task duration estimation and that our heuristics optimize several metrics at the same time while outperforming MCT

Parallel and Distributed Stream Processing: Systems Classification and Specific Issues

Deploying an infrastructure to execute queries on distributed data streams sources requires to identify a scalable and robust solution able to provide results which can be qualified. Last decade, different Data Stream Management Systems have been designed by exploiting new paradigm and technologies to improve performances of solutions facing specific features of data streams and their growing number. However, some tradeoffs are often achieved between performance of the processing, resources consumption and quality of results. This survey 5 suggests an overview of existing solutions among distributed and parallel systems classified according to criteria able to allow readers to efficiently identify relevant existing Distributed Stream Management Systems according to their needs ans resources

New Dynamic Heuristics in the Client-Agent-Server Model

Colloque avec actes et comité de lecture. internationale.International audienceMCT is a widely used heuristic for scheduling tasks onto grid platforms. However, when dealing with many tasks, MCT tends to dramatically delay already mapped task completion time, while scheduling a new task. In this paper we propose heuristics based on two features: the historical trace manager that simulates the environment and the perturbation that defines the impact a new allocated task has on already mapped tasks. Our simulations and experiments on a real environment show that the proposed heuristics outperform MCT

Large-scale parallelism for constraint-based local search: the costas array case study

International audienceWe present the parallel implementation of a constraint-based Local Search algorithm and investigate its performance on several hardware plat-forms with several hundreds or thousands of cores. We chose as the basis for these experiments the Adaptive Search method, an efficient sequential Local Search method for Constraint Satisfaction Problems (CSP). After preliminary experiments on some CSPLib benchmarks, we detail the modeling and solving of a hard combinatorial problem related to radar and sonar applications: the Costas Array Problem. Performance evaluation on some classical CSP bench-marks shows that speedups are very good for a few tens of cores, and good up to a few hundreds of cores. However for a hard combinatorial search problem such as the Costas Array Problem, performance evaluation of the sequential version shows results outperforming previous Local Search implementations, while the parallel version shows nearly linear speedups up to 8,192 cores. The proposed parallel scheme is simple and based on independent multi-walks with no communication between processes during search. We also investigated a cooperative multi-walk scheme where processes share simple information, but this scheme does not seem to improve performance