Towards Measuring and Understanding Performance in Infrastructure- and Function-as-a-Service Clouds

Context. Cloud computing has become the de facto standard for deploying modern software systems, which makes its performance crucial to the efficient functioning of many applications. However, the unabated growth of established cloud services, such as Infrastructure-as-a-Service (IaaS), and the emergence of new services, such as Function-as-a-Service (FaaS), has led to an unprecedented diversity of cloud services with different performance characteristics.Objective. The goal of this licentiate thesis is to measure and understand performance in IaaS and FaaS clouds. My PhD thesis will extend and leverage this understanding to propose solutions for building performance-optimized FaaS cloud applications.Method.\ua0To achieve this goal, quantitative and qualitative research methods are used, including experimental research, artifact analysis, and literature review.Findings.\ua0The thesis proposes a cloud benchmarking methodology to estimate application performance in IaaS clouds, characterizes typical FaaS applications, identifies gaps in literature on FaaS performance evaluations, and examines the reproducibility of reported FaaS performance experiments. The evaluation of the benchmarking methodology yielded promising results for benchmark-based application performance estimation under selected conditions. Characterizing 89 FaaS applications revealed that they are most commonly used for short-running tasks with low data volume and bursty workloads. The review of 112 FaaS performance studies from academic and industrial sources found a strong focus on a single cloud platform using artificial micro-benchmarks and discovered that the majority of studies do not follow reproducibility principles on cloud experimentation.Future Work. Future work will propose a suite of application performance benchmarks for FaaS, which is instrumental for evaluating candidate solutions towards building performance-optimized FaaS applications

Performance Evaluation of Serverless Applications and Infrastructures

Context. Cloud computing has become the de facto standard for deploying modern web-based software systems, which makes its performance crucial to the efficient functioning of many applications. However, the unabated growth of established cloud services, such as Infrastructure-as-a-Service (IaaS), and the emergence of new serverless services, such as Function-as-a-Service (FaaS), has led to an unprecedented diversity of cloud services with different performance characteristics. Measuring these characteristics is difficult in dynamic cloud environments due to performance variability in large-scale distributed systems with limited observability.Objective. This thesis aims to enable reproducible performance evaluation of serverless applications and their underlying cloud infrastructure.Method. A combination of literature review and empirical research established a consolidated view on serverless applications and their performance. New solutions were developed through engineering research and used to conduct performance benchmarking field experiments in cloud environments.Findings. The review of 112 FaaS performance studies from academic and industrial sources found a strong focus on a single cloud platform using artificial micro-benchmarks and discovered that most studies do not follow reproducibility principles on cloud experimentation. Characterizing 89 serverless applications revealed that they are most commonly used for short-running tasks with low data volume and bursty workloads. A novel trace-based serverless application benchmark shows that external service calls often dominate the median end-to-end latency and cause long tail latency. The latency breakdown analysis further identifies performance challenges of serverless applications, such as long delays through asynchronous function triggers, substantial runtime initialization for coldstarts, increased performance variability under bursty workloads, and heavily provider-dependent performance characteristics. The evaluation of different cloud benchmarking methodologies has shown that only selected micro-benchmarks are suitable for estimating application performance, performance variability depends on the resource type, and batch testing on the same instance with repetitions should be used for reliable performance testing.Conclusions. The insights of this thesis can guide practitioners in building performance-optimized serverless applications and researchers in reproducibly evaluating cloud performance using suitable execution methodologies and different benchmark types

D-SPACE4Cloud: Towards Quality-Aware Data Intensive Applications in the Cloud

The last years witnessed a steep rise in data generation worldwide and, consequently, the widespread adoption of software solutions claiming to support data intensive applications. Competitiveness and innovation have strongly benefited from these new platforms and methodologies, and there is a great deal of interest around the new possibilities that Big Data analytics promise to make reality. Many companies currently en- gage in data intensive processes as part of their core businesses; however, fully embracing the data-driven paradigm is still cumbersome, and es- tablishing a production-ready, fine-tuned deployment is time-consuming, expensive, and resource-intensive. This situation calls for novel models and techniques to streamline the process of deployment configuration for Big Data applications. In particular, the focus in this paper is on the rightsizing of Cloud deployed clusters, which represent a cost-effective alternative to installation on premises. We propose a novel tool, inte- grated in a wider DevOps-inspired approach, implementing a parallel and distributed simulation-optimization technique that efficiently and effec- tively explores the space of alternative resource configurations, seeking the minimum cost deployment that satisfies predefined quality of service constraints. The validity and relevance of the proposed solution has been thoroughly validated in a vast experimental campaign including different applications and Big Data platforms

Estimating Cloud Application Performance Based on Micro-Benchmark Profiling

The continuing growth of the cloud computing market has led to an unprecedented diversity of cloud services. To support service selection, micro-benchmarks are commonly used to identify the best performing cloud service. However, it remains unclear how relevant these synthetic micro-benchmarks are for gaining insights into the performance of real-world applications.Therefore, this paper develops a cloud benchmarking methodology that uses micro-benchmarks to profile applications and subsequently predicts how an application performs on a wide range of cloud services. A study with a real cloud provider (Amazon EC2) has been conducted to quantitatively evaluate the estimation model with 38 metrics from 23 micro-benchmarks and 2 applications from different domains. The results reveal remarkably low variability in cloud service performance and show that selected micro-benchmarks can estimate the duration of a scientific computing application with a relative error of less than 10% and the response time of a Web serving application with a relative error between 10% and 20%. In conclusion, this paper emphasizes the importance of cloud benchmarking by substantiating the suitability of micro-benchmarks for estimating application performance in comparison to common baselines but also highlights that only selected micro-benchmarks are relevant to estimate the performance of a particular application

Strategies to Manage Cloud Computing Operational Costs

Information technology (IT) managers worldwide have adopted cloud computing because of its potential to improve reliability, scalability, security, business agility, and cost savings; however, the rapid adoption of cloud computing has created challenges for IT managers, who have reported an estimated 30% wastage of cloud resources. The purpose of this single case study was to explore successful strategies and processes for managing infrastructure operations costs in cloud computing. The sociotechnical systems (STS) approach was the conceptual framework for the study. Semistructured interviews were conducted with 6 IT managers directly involved in cloud cost management. The data were analyzed using a qualitative data-analysis software to identify initial categories and emerging themes, which were refined in alignment with the STS framework. The key themes from the analysis indicated that successful cloud cost management began with assessing the current environment and architecting applications and systems to fit cloud services, using tools for monitoring and reporting, and actively managing costs in alignment with medium- and long-term goals. Findings also indicated that social considerations such as fostering collaboration among all stakeholders, employee training, and skills development were critical for success. The implications for positive social change that derive from effectively managing operational costs include improved financial posture, job stability, and environmental sustainability

Quantifying cloud performance and dependability:Taxonomy, metric design, and emerging challenges

In only a decade, cloud computing has emerged from a pursuit for a service-driven information and communication technology (ICT), becoming a significant fraction of the ICT market. Responding to the growth of the market, many alternative cloud services and their underlying systems are currently vying for the attention of cloud users and providers. To make informed choices between competing cloud service providers, permit the cost-benefit analysis of cloud-based systems, and enable system DevOps to evaluate and tune the performance of these complex ecosystems, appropriate performance metrics, benchmarks, tools, and methodologies are necessary. This requires re-examining old system properties and considering new system properties, possibly leading to the re-design of classic benchmarking metrics such as expressing performance as throughput and latency (response time). In this work, we address these requirements by focusing on four system properties: (i) elasticity of the cloud service, to accommodate large variations in the amount of service requested, (ii) performance isolation between the tenants of shared cloud systems and resulting performance variability, (iii) availability of cloud services and systems, and (iv) the operational risk of running a production system in a cloud environment. Focusing on key metrics for each of these properties, we review the state-of-the-art, then select or propose new metrics together with measurement approaches. We see the presented metrics as a foundation toward upcoming, future industry-standard cloud benchmarks

A Predictive Approach for the Efficient Distribution of Agent-Based Systems on a Hybrid-Cloud

International audienceHybrid clouds are increasingly used to outsource non-critical applications to public clouds. However, the main challenge within such environments, is to ensure a cost-efficient distribution of the systems between the resources that are on/off premises. For Multi Agent Systems (MAS), this challenge is deepened due to irregular workload progress and intensive communication between the agents, which may result in high computing and data transfer costs. Thus, in this paper we propose a generic framework for adaptive cost-efficient deployment of MAS with a special focus on hybrid clouds. The framework is based mainly on the use of a performance evaluation process that consists of simulating various partitioning options to estimate and optimize the overall deployment costs. Further, to cope with the irregular workload changes within a MAS and dynamically adapt its initial deployment, we propose an extended version of the Fiduccia-Mattheyses algorithm (E-FM). The experimental results highlight the efficiency of E-FM and show that an efficient MAS deployment to hybrid clouds depends on various factors such as the cloud providers and their different cost-models, the network state, the used partitioning algorithm, and the initial deployment

Evaluación del benchmarking en la mejora continua de las prácticas empresariales. Revisión sistemática

El informe de revisión sistemática tuvo como objetivo revisar la información existente y proponer actualización de información desde el enfoque de mejora continua de la variable benchmarking aplicada a las áreas de la empresa. La investigación se llevó desde un enfoque cualitativo, diseño de revisión sistemática, se realizó una búsqueda de revistas indexadas desde el 2017 hasta el 2021; de los cuales se aplicó los criterios de inclusión y exclusión , finalmente se incorporaron 31 artículos de revisión. Se obtuvo como resultado que el benchmarking es una técnica que contribuye con la mejora continua, puesto que se trata de conocer las mejores prácticas de otras organizaciones y aplicarlas en las propias. Como conclusiones generales se establece que la aplicación del benchmarking brinda beneficios en los procesos, tecnología, calidad, finanzas y logística, además el éxito del benchmarking depende, sobre todo, de que la empresa acepte que necesita realizar cambios. Finalmente se recomienda a las empresas que desean aplicar el benchmarking, salgan de su zona de confort y estén abiertas a n uevas ideas, buscando el cambio y orientándose hacia la acción; para ello deben apoyarse de un especialista benchmarker, mantener una constancia y disciplina

Change Management Systems for Seamless Evolution in Data Centers

Revenue for data centers today is highly dependent on the satisfaction of their enterprise customers. These customers often require various features to migrate their businesses and operations to the cloud. Thus, clouds today introduce new features at a swift pace to onboard new customers and to meet the needs of existing ones. This pace of innovation continues to grow on super linearly, e.g., Amazon deployed 1400 new features in 2017. However, such a rapid pace of evolution adds complexities both for users and the cloud. Clouds struggle to keep up with the deployment speed, and users struggle to learn which features they need and how to use them. The pace of these evolutions has brought us to a tipping point: we can no longer use rules of thumb to deploy new features, and customers need help to identify which features they need. We have built two systems: Janus and Cherrypick, to address these problems. Janus helps data center operators roll out new changes to the data center network. It automatically adapts to the data center topology, routing, traffic, and failure settings. The system reduces the risk of new deployments for network operators as they can now pick deployment strategies which are less likely to impact users’ performance. Cherrypick finds near-optimal cloud configurations for big data analytics. It adapts allows users to search through the new machine types the clouds are constantly introducing and find ones with a near-optimal performance that meets their budget. Cherrypick can adapt to new big-data frameworks and applications as well as the new machine types the clouds are constantly introducing. As the pace of cloud innovations increases, it is critical to have tools that allow operators to deploy new changes as well as those that would enable users to adapt to achieve good performance at low cost. The tools and algorithms discussed in this thesis help accomplish these goals

BIG DATA и анализ высокого уровня : материалы конференции

В сборнике опубликованы результаты научных исследований и разработок в области BIG DATA and Advanced Analytics для оптимизации IT-решений и бизнес-решений, а также тематических исследований в области медицины, образования и экологии