Themis: A Spot-Market Based Automatic Resource Scaling Framework

International audienceCloud computing brings new provisioning models that offer applications more flexibility and better control over their resource allocations. However these models suffer from the following problem: either they provide limited support for applications demanding quality of service, or they lead to a limited infrastructure utilization. In this paper we propose Themis, a novel resource management system for virtualized infrastructures based on a virtual economy. By limiting the coupling between the applications and the infrastructure through the use of a dynamic resource pricing mechanism, Themis can support diverse types of applications and performance goals while ensuring an efficient resource usage

Towards an end-to-end analysis and prediction system for weather, climate, and marine applications in the Red Sea

Author Posting. © American Meteorological Society, 2021. This article is posted here by permission of American Meteorological Society for personal use, not for redistribution. The definitive version was published in Bulletin of the American Meteorological Society 102(1), (2021): E99-E122, https://doi.org/10.1175/BAMS-D-19-0005.1.The Red Sea, home to the second-longest coral reef system in the world, is a vital resource for the Kingdom of Saudi Arabia. The Red Sea provides 90% of the Kingdom’s potable water by desalinization, supporting tourism, shipping, aquaculture, and fishing industries, which together contribute about 10%–20% of the country’s GDP. All these activities, and those elsewhere in the Red Sea region, critically depend on oceanic and atmospheric conditions. At a time of mega-development projects along the Red Sea coast, and global warming, authorities are working on optimizing the harnessing of environmental resources, including renewable energy and rainwater harvesting. All these require high-resolution weather and climate information. Toward this end, we have undertaken a multipronged research and development activity in which we are developing an integrated data-driven regional coupled modeling system. The telescopically nested components include 5-km- to 600-m-resolution atmospheric models to address weather and climate challenges, 4-km- to 50-m-resolution ocean models with regional and coastal configurations to simulate and predict the general and mesoscale circulation, 4-km- to 100-m-resolution ecosystem models to simulate the biogeochemistry, and 1-km- to 50-m-resolution wave models. In addition, a complementary probabilistic transport modeling system predicts dispersion of contaminant plumes, oil spill, and marine ecosystem connectivity. Advanced ensemble data assimilation capabilities have also been implemented for accurate forecasting. Resulting achievements include significant advancement in our understanding of the regional circulation and its connection to the global climate, development, and validation of long-term Red Sea regional atmospheric–oceanic–wave reanalyses and forecasting capacities. These products are being extensively used by academia, government, and industry in various weather and marine studies and operations, environmental policies, renewable energy applications, impact assessment, flood forecasting, and more.The development of the Red Sea modeling system is being supported by the Virtual Red Sea Initiative and the Competitive Research Grants (CRG) program from the Office of Sponsored Research at KAUST, Saudi Aramco Company through the Saudi ARAMCO Marine Environmental Center at KAUST, and by funds from KAEC, NEOM, and RSP through Beacon Development Company at KAUST

Gestion autonome des ressources et des applications dans un nuage informatique selon une approche fondée sur un marché

Les organisations possédant des infrastructures pour le calcul à haute performance rencontrent des difficultés dans la gestion de leurs ressources. Ces difficultés sont dues au fait que des applications de différents types doivent pouvoir accéder concurremment aux ressources tandis que les utilisateurs peuvent avoir des objectifs de performance variés pour leurs applications. Les nuages informatiques apportent plus de flexibilité et un meilleur contrôle des ressources qui laissent espérer une amélioration de la satisfaction des utilisateurs en terme de qualité de service perçue. Cependant, les solutions de nuage informatique actuelles fournissent un support limité aux utilisateurs pour l'expression ou l'utilisation de politiques de gestion de ressources et elles n'offrent aucun support pour atteindre les objectifs de performance des applications. Dans cette thèse, nous présentons une approche qui aborde ce défi d'une manière unique. Notre approche offre un contrôle des ressources complètement décentralisé en allouant des ressources à travers un marché à pourcentage proportionnel tandis que les applications s'exécutent dans des environnements virtuels autonomes capable d'ajuster la demande de l'application selon les objectifs de performance définis par l'utilisateur. La combinaison de la politique de distribution de la monnaie et de la variation dynamique du prix des ressources assure une utilisation des ressources équitable. Nous avons évalué notre approche en simulation et expérimentalement sur la plate-forme Grid'5000. Nos résultats montrent que notre approche peut permettre la cohabitation des différentes politiques d'utilisation des ressources sur l'infrastructure, tout en améliorant l'utilisation des ressources.Organizations owning HPC infrastructures are facing difficulties in managing their resources. These difficulties come from the need to provide concurrent resource access to different application types while considering that users might have different performance objectives for their applications. Cloud computing brings more flexibility and better resource control, promising to improve the user s satisfaction in terms of perceived Quality of Service. Nevertheless, current cloud solutions provide limited support for users to express or use various resource management policies and they don't provide any support for application performance objectives.In this thesis, we present an approach that addresses this challenge in an unique way. Our approach provides a fully decentralized resource control by allocating resources through a proportional-share market, while applications run in autonomous virtual environments capable of scaling the application demand according to user performance objectives.The combination of currency distribution and dynamic resource pricing ensures fair resource utilization.We evaluated our approach in simulation and on the Grid'5000 testbed. Our results show that our approach can enable the co-habitation of different resource usage policies on the infrastructure, improving resource utilisation.RENNES1-Bibl. électronique (352382106) / SudocSudocFranceF

Abstraction des systèmes informatiques à haute performance pour l'automatisation du déploiement d'applications dynamiques

L'informatique est un formidable levier du développement humain, cependant l'utilisation de systèmes à haute performance reste complexe. En effet, allouer des ressources pour une application, l'installer sur les ressources allouées, et l'exécuter est un processus difficile. Automatiser ce processus, appelé déploiement, permet de mieux séparer les préoccupations des utilisateurs de systèmes, des développeurs d'applications, et des administrateurs d'infrastructures tout en visant de meilleures performances pour les applications. Fondée sur un modèle d'abstraction multi-niveau, la contribution de cette thèse est une architecture de déploiement automatique capable de respecter les contraintes de compatibilité et de faisabilité, inhérentes au déploiement, pour des ressources et des applications parallèles et réparties, hétérogènes, et dynamiques telles que des grappes ou des grilles de calcul et des applications paramétriques, parallèles, ou en flot de travail.Computing is an important lever in human development, however using high performance systems remains complex. Indeed, allocating some resources for an application, installing it on the allocated resources, and executing it is a difficult process. Automating this process, called deployment, allows to better separate the concerns of the system users, of the application developers, and of the infrastructure administrators while aiming at better performances for the applications. Based on a multi-level abstraction model, the contribution of this thesis is an automatic deployment architecture able to honor the compatibility and feasibility constraints, inhere in the deployment, for resources and applications parallel and distributed, heterogeneous, and dynamic such as computing clusters or grids and parametric, parallel, or workflow applications.RENNES1-BU Sciences Philo (352382102) / SudocSudocFranceF

Market-based Autonomous Resource and Application Management in Private Clouds

International audienceHigh Performance Computing (HPC) clouds need to be efficiently shared between selfish tenants having applications with different resource requirements and Service Level Objectives (SLOs). The main difficulty relies on providing concurrent resource access to such tenants while maximizing the resource utilization. To overcome this challenge, we propose Merkat, a market-based SLO-driven cloud platform. Merkat relies on a market-based model specifically designed for on-demand fine-grain resource allocation to maximize resource utilization and it uses a combination of currency distribution and dynamic resource pricing to ensure proper resource distribution among tenants. To meet the tenant’s SLO, Merkat uses autonomous controllers, which apply adaptation policies that: (i) dynamically tune the application’s provisioned CPU and memory per virtual machine in contention periods, or (ii) dynamically change the number of virtual machines. Our evaluation with simulation and on the Grid’5000 testbed shows that Merkat provides flexible support for different application types and SLOs and good tenant satisfaction compared to existing centralized systems, while the infrastructure resource utilization is improved

Merkat: Market-based Autonomous Application and Resource Management in the Cloud

Organizations owning HPC infrastructures are facing difficulties in managing their infrastructures. These difficulties come from the need to provide concurrent resource access to applications with different resource requirements while considering that users might have different performance objectives, or Service Level Objectives (SLOs) for executing them. To address these challenges this paper proposes a market-based SLO-driven cloud platform. This platform relies on a market-based model to allocate resources to applications while taking advantage of cloud flexibility to maximize resource utilization. The combination of currency distribution and dynamic resource pricing ensures fair resource distribution. In the same time, autonomous controllers apply adaptation policies to scale the application resource demand according to user SLOs. The adaptation policies can: (i) dynamically tune the amount of CPU and memory provisioned for the virtual machines in contention periods; (ii) dynamically change the number of virtual machines. We evaluated this proposed platform on the Grid'5000 testbed. Results show that: (i) the platform provides flexible support for different application types and different SLOs; (ii) the platform is capable to provide good user satisfaction achieving acceptable performance degradation compared to existing centralized solutions

Use Cases of Virtualization in the Management of Distributed and Parallel Systems

Efficiently sharing resources between multiple applications that run on the same distributed infrastructure is challenging. These applications have different and possible time varying requirements and in the same time, users demand different quality of service guarantees. However, current resource management systems still make it difficult to specify and meet these requirements. Recently, virtualization technologies have been seen as a useful tool for managing distributed infrastructures. This lead us to survey the use cases of virtualization in distributed computing. We start from giving an overview of virtualization techniques together with what advantages and concerns may come from their use. Then we introduce the issues that may come from managing distributed virtualized infrastructure and we present a taxonomy of representative works that used virtualization as a tool for resource management. Finally, we conclude with an analysis of the advancements done in resource management with the use of virtualization

Market-based Autonomous Application and Resource Management in the Cloud

Version soumise à la revue internationale JPDCOrganizations owning High Performance Computing (HPC) infrastructures are facing difficulties in managing their resources. These difficulties come from the need to provide concurrent resource access to applications with different resource requirements while considering that users are selfish and might have different performance objectives, or Service Level Objectives (SLOs), when executing them. To address these challenges, this paper proposes Merkat, a market-based SLO-driven cloud platform. Merkat relies on a market-based model to allocate resources to ap- plications while taking advantage of virtualization technologies and on-demand provisioning to maximize resource utilization. Merkat’s resource market uses a combination of currency distribu- tion and dynamic resource pricing to ensure proper resource distribution while decentralizing the resource control. In Merkat autonomous controllers apply adaptation policies to scale the appli- cation’s resource demand according to user’s SLO. The adaptation policies can: (i) dynamically tune the amount of CPU and memory provisioned for the virtual machines in contention periods; (ii) dynamically change the number of virtual machines. We evaluated this proposed platform in simulation and on the Grid’5000 testbed. Results show that: (i) Merkat provides flexible suport for different application types and different SLOs; (ii) it increases the resource utilization of the infras- tructure; (iii) and is capable of providing good user satisfaction compared to existing centralized systems.Résumé : Les organisations qui possèdent des infrastructures de calcul à haute performance (HPC) font souvent face à certaines difficultés dans la ges- tion de leurs ressources. En particulier, ces difficultés peuvent provenir du fait que des applications de différents types doivent pouvoir accéder concurrem- ment aux ressources tandis que les utilisateurs peuvent avoir des objectifs de performance (SLOs) variés. Pour résoudre ces difficultés, cet article propose un cadre générique et extensible pour la gestion autonome des applications et l’allocation dynamique des ressources. L’allocation des ressources et l’exécution des applications sont régies par une économie de marché respectant au mieux des objectifs de niveau de service (SLO) tout en tirant parti de la flexibil- ité d’un nuage informatiqueé et en maximisant l’utilisation des ressources. Le marché fixe dynamiquement un prix aux ressources, ce qui, combiné avec une politique de distribution de monnaie entre les utilisateurs, en garantit une utili- sation équitable. Simultanément, des contrôleurs autonomes mettent en œuvre des politiques d’adaptation pour faire évoluer la demande en ressource de leur application en accord avec les objectifs (SLO) fixés par l’utilisateur. Les poli- tiques d’adaptation peuvent : (i) adapter dynamiquement leur demande en terme de CPU et de mémoire pour les machines virtuelles en période de con- tention pour l’obtention de ressources (ii) et changer dynamiquement le nombre de machines virtuelles. Nous avons évalué cette plate-forme par simulation et sur l’infrastructure Grid’5000. Nos résultats ont montré que cette solution: (i) offre un support flexible aux applications de différents types ayant des demandes variés en terme de niveau de service; (ii) augmente l’utilisation des ressources de l’infrastructure; (iii) conduit à une meilleure satisfaction des utilisateurs par rapport aux solutions centralisées existantes

Merkat: A Market-based SLO-driven Cloud Platform

International audienceCloud computing brings new service models that offer users more flexibility and better resource control promising to improve the user's satisfaction in terms of Quality of Service. However, current cloud solutions, in particular above the Infrastructure-as-a-Service layer, don't provide automated solutions that address the Quality of Service users may want for their applications. In this paper we present Merkat, a platform that is capable to dynamically share cloud resources between the different application types using a market-based resource allocation mechanism. Merkat provides fair and maximum resource utilization by allocating fine-grained resource amounts to users proportionally to their communicated value. In the same time, autonomous controllers provide SLO support by applying elasticity rules to scale the application demand according to user performance objectives. Our experiments show that Merkat is able to efficiently react to changes in application load and user requirements, leading to better user satisfaction

Merkat: A Market-based SLO-driven Cloud Platform

International audienceCloud computing brings new service models that offer users more flexibility and better resource control promising to improve the user's satisfaction in terms of Quality of Service. However, current cloud solutions, in particular above the Infrastructure-as-a-Service layer, don't provide automated solutions that address the Quality of Service users may want for their applications. In this paper we present Merkat, a platform that is capable to dynamically share cloud resources between the different application types using a market-based resource allocation mechanism. Merkat provides fair and maximum resource utilization by allocating fine-grained resource amounts to users proportionally to their communicated value. In the same time, autonomous controllers provide SLO support by applying elasticity rules to scale the application demand according to user performance objectives. Our experiments show that Merkat is able to efficiently react to changes in application load and user requirements, leading to better user satisfaction