Cloud WorkBench - Infrastructure-as-Code Based Cloud Benchmarking

To optimally deploy their applications, users of Infrastructure-as-a-Service clouds are required to evaluate the costs and performance of different combinations of cloud configurations to find out which combination provides the best service level for their specific application. Unfortunately, benchmarking cloud services is cumbersome and error-prone. In this paper, we propose an architecture and concrete implementation of a cloud benchmarking Web service, which fosters the definition of reusable and representative benchmarks. In distinction to existing work, our system is based on the notion of Infrastructure-as-Code, which is a state of the art concept to define IT infrastructure in a reproducible, well-defined, and testable way. We demonstrate our system based on an illustrative case study, in which we measure and compare the disk IO speeds of different instance and storage types in Amazon EC2

Towards Measuring and Understanding Performance in Infrastructure- and Function-as-a-Service Clouds

Context. Cloud computing has become the de facto standard for deploying modern software systems, which makes its performance crucial to the efficient functioning of many applications. However, the unabated growth of established cloud services, such as Infrastructure-as-a-Service (IaaS), and the emergence of new services, such as Function-as-a-Service (FaaS), has led to an unprecedented diversity of cloud services with different performance characteristics.Objective. The goal of this licentiate thesis is to measure and understand performance in IaaS and FaaS clouds. My PhD thesis will extend and leverage this understanding to propose solutions for building performance-optimized FaaS cloud applications.Method.\ua0To achieve this goal, quantitative and qualitative research methods are used, including experimental research, artifact analysis, and literature review.Findings.\ua0The thesis proposes a cloud benchmarking methodology to estimate application performance in IaaS clouds, characterizes typical FaaS applications, identifies gaps in literature on FaaS performance evaluations, and examines the reproducibility of reported FaaS performance experiments. The evaluation of the benchmarking methodology yielded promising results for benchmark-based application performance estimation under selected conditions. Characterizing 89 FaaS applications revealed that they are most commonly used for short-running tasks with low data volume and bursty workloads. The review of 112 FaaS performance studies from academic and industrial sources found a strong focus on a single cloud platform using artificial micro-benchmarks and discovered that the majority of studies do not follow reproducibility principles on cloud experimentation.Future Work. Future work will propose a suite of application performance benchmarks for FaaS, which is instrumental for evaluating candidate solutions towards building performance-optimized FaaS applications

Performance Evaluation of Serverless Applications and Infrastructures

Context. Cloud computing has become the de facto standard for deploying modern web-based software systems, which makes its performance crucial to the efficient functioning of many applications. However, the unabated growth of established cloud services, such as Infrastructure-as-a-Service (IaaS), and the emergence of new serverless services, such as Function-as-a-Service (FaaS), has led to an unprecedented diversity of cloud services with different performance characteristics. Measuring these characteristics is difficult in dynamic cloud environments due to performance variability in large-scale distributed systems with limited observability.Objective. This thesis aims to enable reproducible performance evaluation of serverless applications and their underlying cloud infrastructure.Method. A combination of literature review and empirical research established a consolidated view on serverless applications and their performance. New solutions were developed through engineering research and used to conduct performance benchmarking field experiments in cloud environments.Findings. The review of 112 FaaS performance studies from academic and industrial sources found a strong focus on a single cloud platform using artificial micro-benchmarks and discovered that most studies do not follow reproducibility principles on cloud experimentation. Characterizing 89 serverless applications revealed that they are most commonly used for short-running tasks with low data volume and bursty workloads. A novel trace-based serverless application benchmark shows that external service calls often dominate the median end-to-end latency and cause long tail latency. The latency breakdown analysis further identifies performance challenges of serverless applications, such as long delays through asynchronous function triggers, substantial runtime initialization for coldstarts, increased performance variability under bursty workloads, and heavily provider-dependent performance characteristics. The evaluation of different cloud benchmarking methodologies has shown that only selected micro-benchmarks are suitable for estimating application performance, performance variability depends on the resource type, and batch testing on the same instance with repetitions should be used for reliable performance testing.Conclusions. The insights of this thesis can guide practitioners in building performance-optimized serverless applications and researchers in reproducibly evaluating cloud performance using suitable execution methodologies and different benchmark types

Performance Benchmarking of Infrastructure-as-a-Service (IaaS) Clouds with Cloud WorkBench (Tutorial)

The continuing growth of the cloud computing market has led to an unprecedented diversity of cloud services with different performance characteristics. To support service selection, researchers and practitioners conduct cloud performance benchmarking by measuring and objectively comparing the performance of different providers and configurations (e.g., instance types in different data center regions). In this tutorial, we demonstrate how to write performance tests for IaaS clouds using the Web-based benchmarking tool Cloud WorkBench (CWB). We will motivate and introduce benchmarking of IaaS cloud in general, demonstrate the execution of a simple benchmark in a public cloud environment, summarize the CWB tool architecture, and interactively develop and deploy a more advanced benchmark together with the participants

Function-as-a-Service Performance Evaluation: A Multivocal Literature Review

Function-as-a-Service (FaaS) is one form of the serverless cloud computing paradigm and is defined through FaaS platforms (e.g., AWS Lambda) executing event-triggered code snippets (i.e., functions). Many studies that empirically evaluate the performance of such FaaS platforms have started to appear but we are currently lacking a comprehensive understanding of the overall domain. To address this gap, we conducted a multivocal literature review (MLR) covering 112 studies from academic (51) and grey (61) literature. We find that existing work mainly studies the AWS Lambda platform and focuses on micro-benchmarks using simple functions to measure CPU speed and FaaS platform overhead (i.e., container cold starts). Further, we discover a mismatch between academic and industrial sources on tested platform configurations, find that function triggers remain insufficiently studied, and identify HTTP API gateways and cloud storages as the most used external service integrations. Following existing guidelines on experimentation in cloud systems, we discover many flaws threatening the reproducibility of experiments presented in the surveyed studies. We conclude with a discussion of gaps in literature and highlight methodological suggestions that may serve to improve future FaaS performance evaluation studies.Comment: improvements including postprint update

Software Microbenchmarking in the Cloud. How Bad is it Really?

Rigorous performance engineering traditionally assumes measuring on bare-metal environments to control for as many confounding factors as possible. Unfortunately, some researchers and practitioners might not have access, knowledge, or funds to operate dedicated performance-testing hardware, making public clouds an attractive alternative. However, shared public cloud environments are inherently unpredictable in terms of the system performance they provide. In this study, we explore the effects of cloud environments on the variability of performance test results and to what extent slowdowns can still be reliably detected even in a public cloud. We focus on software microbenchmarks as an example of performance tests and execute extensive experiments on three different well-known public cloud services (AWS, GCE, and Azure) using three different cloud instance types per service. We also compare the results to a hosted bare-metal offering from IBM Bluemix. In total, we gathered more than 4.5 million unique microbenchmarking data points from benchmarks written in Java and Go. We find that the variability of results differs substantially between benchmarks and instance types (by a coefficient of variation from 0.03% to > 100%). However, executing test and control experiments on the same instances (in randomized order) allows us to detect slowdowns of 10% or less with high confidence, using state-of-the-art statistical tests (i.e., Wilcoxon rank-sum and overlapping bootstrapped confidence intervals). Finally, our results indicate that Wilcoxon rank-sum manages to detect smaller slowdowns in cloud environments

Performance Benchmarking of Infrastructure-as-a-Service (IaaS) Clouds with CloudWorkBench

The continuing growth of the cloud computing market has led to an unprecedented diversity of cloud services with different performance characteristics. To support service selection, researchers and practitioners conduct cloud performance benchmarking by measuring and objectively comparing the performance of different providers and configurations (e.g., instance types in different data center regions). In this tutorial, we demonstrate how to write performance tests for IaaS clouds using the Web-based benchmarking tool Cloud WorkBench (CWB). We will motivate and introduce benchmarking of IaaS cloud in general, demonstrate the execution of a simple benchmark in a public cloud environment, summarize the CWB tool architecture, and interactively develop and deploy a more advanced benchmark together with the participants

L’État, la formation et le contrôle des associations dans l’Allemagne du XIXe siècle

1. L’État et les pouvoirs intermédiaires dans les développements contemporains Dans cette présentation, il s’agira pour nous de suivre deux lignes étroitement dépendantes l’une de l’autre des relations modernes entre le pouvoir d’État et les associations. D’une part l’attitude de l’État face à la structuration interne de la société, dont dépend également sa position à l’égard d’associations libres et de celles qu’il dirige, d’autre part les différences, certes toujours délicates à déterminer,..

Auswirkungen atmogener Stickstoffeinträge auf die Kohlenstoff- und Stickstoffdynamik unterschiedlich stark stickstoffbelasteter Waldböden

Das Ziel der Arbeit war die Untersuchung der Kohlenstoff- und Stickstoffdynamik in Waldböden unter dem Einfluß atmogener Stickstoffeinträge. Dabei wurden biotische und abiotische Schlüsselprozesse zweier unterschiedlich stark stickstoffbelasteter Waldböden des Nordwest- und Nordostdeutschen Tieflandes (Kreinitz und Thülsfeld) analysiert und miteinander verglichen. Um detaillierte Informationen über potentielle Kohlenstoff- und Stickstoffumsätze zu erhalten, wurden die untersuchten Böden sowohl horizont- als auch tiefenstufenbezogen stratifiziert. Über einen Zeitraum von 14 Wochen wurden sämtliche Kohlenstoff- und Stickstoffeinträge sowie -austräge zwischen den einzelnen Tiefenstufen beobachtet und analysiert. Nach Beendigung der Versuche wurden die einzelnen Bodenhorizonte mit spezifischen Kohlenstoff- und Stickstoffextraktionsverfahren aufgeschlossen. Mit Hilfe dieser Versuchsanordnung war es möglich, eine vollständige tiefenstufenbezogene Kohlenstoff- und Stickstoffbilanzierung für den Kreinitzer und Thülsfelder Boden zu erstellen. Zur Prüfung der Übertragbarkeit der Laborergebnisse auf aktuelle Standortbedingungen wurde ein Langzeitfeldversuch (ein Jahr) mit einer ähnlichen Versuchsanordnung am Standort Kreinitz durchgeführt. Folgende Ergebnisse wurden in dieser Arbeit erzielt: 1. Der TOC-Gehalt war der Hauptparameter, in dem sich der stark stickstoffbelastete Standort Thülsfeld von dem weniger stark stickstoffbelasteten Standort Kreinitz zu Beginn der Untersuchung deutlich unterschied. 2. Der KCl-extrahierbare mineralische Stickstoff (NminKCl) reagierte an beiden Versuchs-standorten am stärksten auf die Stickstoffdüngung und wurde mit Hilfe der Diskriminanzanalyse als Hauptsensitivitätsparameter für Stickstoffeinträge im Boden ermittelt. 3. Die Kreinitzer organische Auflage reagierte in bezug auf die mittelfristig umsetzbare Kohlenstoff- und Stickstofffraktion (Chwe, Nhwe) deutlich stärker auf die N-Einträge als die Thülsfelder Auflage. Infolgedessen besitzt die Kreinitzer Auflage ein höheres Kohlenstoff- und Stickstoffmobilisierungspotential, die hochkomplexen organischen C- und N-Verbindungen in weniger komplexe Verbindungen abzubauen. 4. Der Thülsfelder Mineralboden 5. besitzt aufgrund der höheren Anteile des Nhwe am TN ein höheres Mobilisierungspotential für den kurz- und mittelfristig umsetzbaren Stickstoff (Nhwe) als der Kreinitzer Mineralboden. 6. Die für die Mikroorganismen verfügbaren Kohlenstoffquellen scheinen sich durch die historischen und die zusätzlich simulierten Stickstoffeinträge vom POC in der Sand-Braunerde zum WSOC und CKCl im Sand-Podsol zu verschieben. 7. Als verfügbare Stickstoffquelle nutzen die Mikroorganismen am Standort Kreinitz den PN, während sich für den Standort Thülsfeld keine eindeutige Stickstofffraktion als Hauptnahrungsquelle ermitteln ließ. 8. Sowohl in den Laborversuchen als auch im Freilandversuch führten die Stickstoffeinträge zur Verengung nahezu aller C/N-Verhältnisse in den untersuchten Extraktionsverfahren