Algorithmes d’ordonnancement avec contrainte de budget pour l’exécution de workflows scientifiques avec des tâches de durée stochastique sur plates-formes de type IaaS cloud

This report, which is an update of [5], introduces several budget-aware algorithms to deploy scientific workflows on IaaS Cloud platforms, where users can request Virtual Machines (VMs) of different types, each with specific cost and speed parameters. We use a realistic application/platform model with stochastic task weights, and VMs communicating through a datacenter. We extend two well-known algorithms, Min-Min and HEFT, and make scheduling decisions based upon machine availability and available budget. During the mapping process, the budget-aware algorithms make conservative assumptions to avoid exceeding the initial budget; we further improve our results with refined versions that aim at re-scheduling some tasks onto faster VMs, thereby spending any budget fraction leftover by the first allocation. These refined variants are much more time-consuming than the former algorithms, so there is a trade-off to find in terms of scalability. We report an extensive set of simulations with workflows from the Pegasus benchmark suite. Most of the time our budget-aware algorithms succeed in achieving efficient makespans while enforcing the given budget, and despite the uncertainty inCe rapport, qui est une mise à jour du rapport [5], présente plusieurs algorithmes prenant en compte le budget pour déployer des workflows scientifiques sur des plateformes de Cloud de type IaaS, sur lesquelles les utilisateurs peuvent utiliser des machines virtuelles (ou Virtual Machines, VMs) de différents types, ces dernières étant caractérisées par un coût et une vitesse qui leur sont propres. Nous utilisons un modèle de plateforme et de workflow réalistes avec des tâches de taille stochastique et des VMs communiquant par le biais d’un datacenter.Nous étendons deux algorithmes connus, HEFT et Min-Min, et effectuons l’ordonnancement en nous basant à la fois sur la disponibilité des machines et le budget disponible. Pendant le processus d’attribution des VMs aux tâches, les algorithmes prenant le budget en comptese basent sur des hypothèses conservatives afin d’éviter de dépasser le budget initial ; nous améliorons nos résultats en proposant des algorithmes raffinant ces solutions en tentant de ré-assigner certaines taches à des VMs plus rapides, en utilisant pour ce faire la part de budgetrestant suite à l’ordonnancement initial. Ces versions raffinées demandent plus de temps que les algorithmes proposés plus tôt, il y a donc un compromis à faire en termes de scalabilité. Nous présentons un vaste ensemble de simulations effectuées sur des workflows obtenus àl’aide d’un logiciel de benchmark de Pegasus. Les algorithmes prenant en compte le budget réussissent en général à obtenir des makespans efficaces tout en respectant le budget accordé, et ce malgré l’incertitude concernant la taille des tâches. Nous les comparons également àdeux compétiteurs, BDT and CG/CG+

Budget-aware Static Scheduling of Stochastic Workflows with DIET

International audiencePrevious work has introduced a Cloud platform model and budget-aware static algorithms to schedule stochastic workflows on such platforms. In this paper, we compare the performance of these algorithms obtained via simulation and via execution on an actual platform, Grid'5000. We focus on DIET, a widely used workflow engine, and detail the extensions that were implemented to conduct the comparison. We also detail additional code that we made available in order to automate and to ease the reproducibility of such experiments

Budget-aware workflow scheduling with DIET

DIET is a widely used workflow engine capable of scheduling workflows dynamically. This feature enables the engine to react to various events occurring during execution. However, a static schedule calculated before the submission of the workflow can provide many optimization opportunities, in particular when budget constraints should be enforced. In this paper, we expand DIET by adding a static scheduling functionality. To validate this new functionality on a realistic test case, we compare the behavior of budgetaware algorithms designed in [5], both in simulation and on an actual platform, Grid’5000. The obtained results are very similar, showing that the benefits provided by the scheduling algorithms are effective, both in simulation and in a real execution.DIET est un moteur de workflow très utilisé, capable d’ordonnancer des workflows de manière dynamique. Cette fonctionnalité lui permet de réagir aux divers événements survenant au cours de l’exécution des workflows. Cependant, un ordonnancement statique calculé avant la soumission du workflow peut offrir de nombreuses opportunités d’optimisation, en particulier dans le cas d’une contrainte de budget. Dans ce rapport, nous étendons les fonctionnalités de DIET en y ajoutant la capacité de faire des ordonnancements statiques. Pour valider ce nouveau module avec un cas d’étude réaliste, nous comparons le comportement d’algorithmes présentés dans [5], à la fois par des simulations et sur une vraie plateforme, Grid’5000. Les résultats obtenus sont très similaires, montrant que les avantages apportés par les algorithmes d’ordonnancement sont réels, que ce soit dans le cadre d’une simulation ou d’une vraie exécution

Optimized task scheduling based on hybrid symbiotic organisms search algorithms for cloud computing environment

In Cloud Computing model, users are charged according to the usage of resources and desired Quality of Service (QoS). Task scheduling algorithms are responsible for specifying adequate set of resources to execute user applications in the form of tasks, and schedule decisions of task scheduling algorithms are based on QoS requirements defined by the user. Task scheduling problem is an NP-Complete problem, due to the NP-Complete nature of task scheduling problems and huge search space presented by large scale problem instances, many of the existing solution algorithms incur high computational complexity and cannot effectively obtain global optimum solutions. Recently, Symbiotic Organisms Search (SOS) has been applied to various optimization problems and results obtained were found to be competitive with state-of-the-art metaheuristic algorithms. However, similar to the case other metaheuristic optimization algorithms, the efficiency of SOS algorithm deteriorates as the size of the search space increases. Moreover, SOS suffers from local optima entrapment and its static control parameters cannot maintain a balance between local and global search. In this study, Cooperative Coevolutionary Constrained Multiobjective Symbiotic Organisms Search (CC-CMSOS), Cooperative Coevolutionary Constrained Multi-objective Memetic Symbiotic Organisms Search (CC-CMMSOS), and Cooperative Coevolutionary Constrained Multi-objective Adaptive Benefit Factor Symbiotic Organisms Search (CC-CMABFSOS) algorithms are proposed to solve constrained multi-objective large scale task scheduling optimization problem on IaaS cloud computing environment. To address the issue of scalability, the concept of Cooperative Coevolutionary for enhancing SOS named CC-CMSOS make SOS more efficient for solving large scale task scheduling problems. CC-CMMSOS algorithm further improves the performance of SOS algorithm by hybridizing with Simulated Annealing (SA) to avoid entrapment in local optima for global convergence. Finally, CC-CMABFSOS algorithm adaptively turn SOS control parameters to balance the local and global search procedure for faster convergence speed. The performance of the proposed CC-CMSOS, CC-CMMSOS, and CC-CMABFSOS algorithms are evaluated on CloudSim simulator, using both standard workload traces and synthesized workloads for larger problem instances of up to 5000. Moreover, CC-CMSOS, CC-CMMSOS, and CC-CMABFSOS algorithms are compared with multi-objective optimization algorithms, namely, EMS-C, ECMSMOO, and BOGA. The CC-CMSOS, CC-CMMSOS, and CC-CMABFSOS algorithms obtained significant improved optimal trade-offs between execution time (makespan) and financial cost (cost) while meeting deadline constraints with no computational overhead. The performance improvements obtained by the proposed algorithms in terms of hypervolume ranges from 8.72% to 37.95% across the workloads. Therefore, the proposed algorithms have potentials to improve the performance of QoS delivery

Scheduling independent stochastic tasks under deadline and budget constraints

International audienceThis paper discusses scheduling strategies for the problem of maximizing the expected number of tasks that can be executed on a cloud platform within a given budget and under a deadline constraint. The execution times of tasks follow IID probability laws. The main questions are how many processors to enroll and whether and when to interrupt tasks that have been executing for some time. We provide complexity results and an asymptotically optimal strategy for the problem instance with discrete probability distributions and without deadline. We extend the latter strategy for the general case with continuous distributions and a deadline and we design an efficient heuristic which is shown to outperform standard approaches when running simulations for a variety of useful distribution laws

Provendo robustez a escalonadores de workflows sensíveis às incertezas da largura de banda disponível

Orientadores: Edmundo Roberto Mauro Madeira, Luiz Fernando BittencourtTese (doutorado) - Universidade Estadual de Campinas, Instituto de ComputaçãoResumo: Para que escalonadores de aplicações científicas modeladas como workflows derivem escalonamentos eficientes em nuvens híbridas, é necessário que se forneçam, além da descrição da demanda computacional desses aplicativos, as informações sobre o poder de computação dos recursos disponíveis, especialmente aqueles dados relacionados com a largura de banda disponível. Entretanto, a imprecisão das ferramentas de medição fazem com que as informações da largura de banda disponível fornecida aos escalonadores difiram dos valores reais que deveriam ser considerados para se obter escalonamentos quase ótimos. Escalonadores especialmente projetados para nuvens híbridas simplesmente ignoram a existência de tais imprecisões e terminam produzindo escalonamentos enganosos e de baixo desempenho, o que os tornam sensíveis às informações incertas. A presente Tese introduz um procedimento pró-ativo para fornecer um certo nível de robustez a escalonamentos derivados de escalonadores não projetados para serem robustos frente às incertezas decorrentes do uso de informações imprecisas dadas por ferramentas de medições de rede. Para tornar os escalonamentos sensíveis às incertezas em escalonamentos robustos às essas imprecisões, o procedimento propõe um refinamento (uma deflação) das estimativas da largura de banda antes de serem utilizadas pelo escalonador não robusto. Ao propor o uso de estimativas refinadas da largura de banda disponível, escalonadores inicialmente sensíveis às incertezas passaram a produzir escalonamentos com um certo nível de robustez às essas imprecisões. A eficácia e a eficiência do procedimento proposto são avaliadas através de simulação. Comparam-se, portanto, os escalonamentos gerados por escalonadores que passaram a usar o procedimento proposto com aqueles produzidos pelos mesmos escalonadores mas sem aplicar esse procedimento. Os resultados das simulações mostram que o procedimento proposto é capaz de prover robustez às incertezas da informação da largura de banda a escalonamentos derivados de escalonardes não robustos às tais incertezas. Adicionalmente, esta Tese também propõe um escalonador de aplicações científicas especialmente compostas por um conjunto de workflows. A novidade desse escalonador é que ele é flexível, ou seja, permite o uso de diferentes categorias de funções objetivos. Embora a flexibilidade proposta seja uma novidade no estado da arte, esse escalonador também é sensível às imprecisões da largura de banda. Entretanto, o procedimento mostrou-se capaz de provê-lo de robustez frente às tais incertezas. É mostrado nesta Tese que o procedimento proposto aumentou a eficácia e a eficiência de escalonadores de workflows não robustos projetados para nuvens híbridas, já que eles passaram a produzir escalonamentos com um certo nível de robustez na presença de estimativas incertas da largura de banda disponível. Dessa forma, o procedimento proposto nesta Tese é uma importante ferramenta para aprimorar os escalonadores sensíveis às estimativas incertas da banda disponível especialmente projetados para um ambiente computacional onde esses valores são imprecisos por natureza. Portanto, esta Tese propõe um procedimento que promove melhorias nas execuções de aplicações científicas em nuvens híbridasAbstract: To derive efficient schedules for the tasks of scientific applications modelled as workflows, schedulers need information on the application demands as well as on the resource availability, especially those regarding the available bandwidth. However, the lack of precision of bandwidth estimates provided by monitoring/measurement tools should be considered by the scheduler to achieve near-optimal schedules. Uncertainties of available bandwidth can be a result of imprecise measurement and monitoring network tools and/or their incapacity of estimating in advance the real value of the available bandwidth expected for the application during the scheduling step of the application. Schedulers specially designed for hybrid clouds simply ignore the inaccuracies of the given estimates and end up producing non-robust, low-performance schedules, which makes them sensitive to the uncertainties stemming from using these networking tools. This thesis introduces a proactive procedure to provide a certain level of robustness for schedules derived from schedulers that were not designed to be robust in the face of uncertainties of bandwidth estimates stemming from using unreliable networking tools. To make non-robust schedulers into robust schedulers, the procedure applies a deflation on imprecise bandwidth estimates before being used as input to non-robust schedulers. By proposing the use of refined (deflated) estimates of the available bandwidth, non-robust schedulers initially sensitive to these uncertainties started to produce robust schedules that are insensitive to these inaccuracies. The effectiveness and efficiency of the procedure in providing robustness to non-robust schedulers are evaluated through simulation. Schedules generated by induced-robustness schedulers through the use of the procedure is compared to that of produced by sensitive schedulers. In addition, this thesis also introduces a flexible scheduler for a special case of scientific applications modelled as a set of workflows grouped into ensembles. Although the novelty of this scheduler is the replacement of objective functions according to the user's needs, it is still a non-robust scheduler. However, the procedure was able to provide the necessary robustness for this flexible scheduler be able to produce robust schedules under uncertain bandwidth estimates. It is shown in this thesis that the proposed procedure enhanced the robustness of workflow schedulers designed especially for hybrid clouds as they started to produce robust schedules in the presence of uncertainties stemming from using networking tools. The proposed procedure is an important tool to furnish robustness to non-robust schedulers that are originally designed to work in a computational environment where bandwidth estimates are very likely to vary and cannot be estimated precisely in advance, bringing, therefore, improvements to the executions of scientific applications in hybrid cloudsDoutoradoCiência da ComputaçãoDoutor em Ciência da Computação2012/02778-6FAPES

Online Multi-User Workflow Scheduling Algorithm for Fairness and Energy Optimization

International audienceThis article tackles the problem of scheduling multiuser scientific workflows with unpredictable random arrivals and uncertain task execution times in a Cloud environment from the Cloud provider point of view. The solution consists in a deadline sensitive online algorithm, named NEARDEADLINE, that optimizes two metrics: the energy consumption and the fairness between users. Scheduling workflows in a private Cloud environment is a difficult optimization problem as capacity constraints must be fulfilled additionally to dependencies constraints between tasks of the workflows. Furthermore, NEARDEADLINE is built upon a new workflow execution platform. As far as we know no existing work tries to combine both energy consumption and fairness metrics in their optimization problem. The experiments conducted on a real infrastructure (clusters of Grid'5000) demonstrate that the NEARDEADLINE algorithm offers real benefits in reducing energy consumption, and enhancing user fairness

Resource-Constrained Scheduling of Stochastic Tasks With Unknown Probability Distribution

This work introduces scheduling strategies to maximize the expected numberof independent tasks that can be executed on a cloud platform within a given budgetand under a deadline constraint. Task execution times are not known before execution;instead, the only information available to the scheduler is that they obey some (unknown)probability distribution. The scheduler needs to acquire some information before decidingfor a cutting threshold: instead of allowing all tasks to run until completion, one maywant to interrupt long-running tasks at some point. In addition, the cutting thresholdmay be reevaluated as new information is acquired when the execution progresses further.This works presents several strategies to determine a good cutting threshold, and to decidewhen to re-evaluate it. In particular, we use the Kaplan-Meier estimator to account fortasks that are still running when making a decision. The efficiency of our strategies isassessed through an extensive set of simulations with various budget and deadline values,and ranging over 14 probability distributions.Ce travail présente des stratégies d’ordonnancement permettant de maximiser le nombre attendu de tâches indépendantes pouvant être exécutées sur une plateforme de type cloud avec un budget donné et une contrainte de date limite. Le temps d’exécution des tâches est inconnu, on sait seulement qu’ils obéissent à une distribution de probabilité (inconnue). L’ordonnanceur peut décider à tout moment d’interrompre l’exécution d’une tâche (longue) en cours d’exécution et d’en lancer une nouvelle, mais le budget déjà utilisé pour la tâche interrompue est perdu. Le seuil d’interruption d’une tâche peut être recalculé au fur et à mesure que l’exécution progresse globalement. Ce travail présente plusieurs stratégies pour déterminer un bon seuil d’interruption, et pour décider quand le ré-évaluer. Nous utilisons l’estimateur de Kaplan-Meier pour prendre en compte les tâches en cours d’exécution au moment où la décision est prise. L’efficacité de nos stratégies est évaluée via un vaste ensemble de simulations, avec diverses valeurs de budget et de date limite, et portant sur 14 distributions de probabilit

Scientific Workflow Scheduling for Cloud Computing Environments

The scheduling of workflow applications consists of assigning their tasks to computer resources to fulfill a final goal such as minimizing total workflow execution time. For this reason, workflow scheduling plays a crucial role in efficiently running experiments. Workflows often have many discrete tasks and the number of different task distributions possible and consequent time required to evaluate each configuration quickly becomes prohibitively large. A proper solution to the scheduling problem requires the analysis of tasks and resources, production of an accurate environment model and, most importantly, the adaptation of optimization techniques. This study is a major step toward solving the scheduling problem by not only addressing these issues but also optimizing the runtime and reducing monetary cost, two of the most important variables. This study proposes three scheduling algorithms capable of answering key issues to solve the scheduling problem. Firstly, it unveils BaRRS, a scheduling solution that exploits parallelism and optimizes runtime and monetary cost. Secondly, it proposes GA-ETI, a scheduler capable of returning the number of resources that a given workflow requires for execution. Finally, it describes PSO-DS, a scheduler based on particle swarm optimization to efficiently schedule large workflows. To test the algorithms, five well-known benchmarks are selected that represent different scientific applications. The experiments found the novel algorithms solutions substantially improve efficiency, reducing makespan by 11% to 78%. The proposed frameworks open a path for building a complete system that encompasses the capabilities of a workflow manager, scheduler, and a cloud resource broker in order to offer scientists a single tool to run computationally intensive applications