Quantifying cloud performance and dependability:Taxonomy, metric design, and emerging challenges

In only a decade, cloud computing has emerged from a pursuit for a service-driven information and communication technology (ICT), becoming a significant fraction of the ICT market. Responding to the growth of the market, many alternative cloud services and their underlying systems are currently vying for the attention of cloud users and providers. To make informed choices between competing cloud service providers, permit the cost-benefit analysis of cloud-based systems, and enable system DevOps to evaluate and tune the performance of these complex ecosystems, appropriate performance metrics, benchmarks, tools, and methodologies are necessary. This requires re-examining old system properties and considering new system properties, possibly leading to the re-design of classic benchmarking metrics such as expressing performance as throughput and latency (response time). In this work, we address these requirements by focusing on four system properties: (i) elasticity of the cloud service, to accommodate large variations in the amount of service requested, (ii) performance isolation between the tenants of shared cloud systems and resulting performance variability, (iii) availability of cloud services and systems, and (iv) the operational risk of running a production system in a cloud environment. Focusing on key metrics for each of these properties, we review the state-of-the-art, then select or propose new metrics together with measurement approaches. We see the presented metrics as a foundation toward upcoming, future industry-standard cloud benchmarks

Elasticidade em cloud computing: conceito, estado da arte e novos desafios

A elasticidade é sem dúvida uma das características mais marcantes da computação em nuvem, sendo um diferencial desse tipo de sistema distribuído em relação a outros como grades computacionais e peer-to-peer. Com base nos paradigmas de computação sobre demanda e pague-pelo-que-use, é possível dinamicamente aumentar ou diminuir instâncias de máquinas virtuais e/ou nós de computação, bem como aplicar reconfigurações de percentagem de CPU, memória e largura de banda de rede relativos a um serviço em nuvem. Além dos evidentes benefícios de custo e desempenho para o usuário, o provedor da nuvem também tem a vantagem de oferecer um melhor uso dos recursos perante seus usuários. Nesse contexto, esse artigo apresenta o estado-da-arte na área de elasticidade em nuvem, enfatizando desde a abordagem padrão que usa transações Web até iniciativas para a computação de alto desempenho. Ainda, o artigo discute sobre métricas para ativação da elasticidade, o seu nível de atuação (SaaS, PaaS ou IaaS), bem como a interface de uso (sem intervenção do usuário, linha de comando, ferramenta gráfica ou diretivas de programação). Para fins de experimentação, um estudo de caso do emprego da elasticidade em aplicações de alto desempenho sobre o middleware OpenNebula é apresentado e discutido. Por fim, o artigo aponta os desafios na área e oportunidades de pesquisa, tanto no cunho das nuvens privadas quanto públicas

Elasticity Measurement in CaaS Environments - Extending the Existing BUNGEE Elasticity Benchmark to AWS\u27s Elastic Container Service

Rapid elasticity and automatic scaling are core concepts of most current cloud computing systems. Elasticity describes how well and how fast cloud systems adapt to increases and decreases in workload. In parallel, software architectures are moving towards employing containerised microservices running on systems managed by container orchestration platforms. Cloud users who employ such container-based systems may want to compare the elasticity of different systems or system settings to ensure rapid elasticity and maintain service level objectives while avoiding over-provisioning. Previous research has established a variety of metrics to measure elasticity. Some existing benchmark tools are designed to measure elasticity in “Infrastructure as a Service” (IaaS) systems, but no research exists to date for measuring elasticity in systems based on containers and container orchestration. In this dissertation, an existing benchmark designed for IaaS systems, the BUNGEE benchmark developed at the University of Würzburg, was extended to be applicable to Amazon’s Elastic Container Service, a container-based cloud system. An experiment was conducted to test if the extension of the BUNGEE benchmark described in this dissertation delivers reproducible results and is therefore valid. For validation, the crucial phase of the benchmark - the system analysis phase - was run 32 times. It was established with statistical tests if the results vary by more than the acceptable level. Results indicate that there is some amount of variability, but it does not exceed the acceptable level and is consistent with the amount of performance variability encountered by other researchers in Amazon’s cloud systems. Therefore, it is concluded that the BUNGEE benchmark is likely applicable to container-based cloud systems. However, some parameters and configuration settings specific to container orchestration systems were identified that could impede reproducibility of results and should be considered in future experiments