| 1. |
- Carver, Jeffrey C., et al.
(författare)
-
(Research) Insights for Serverless Application Engineering
- 2021
-
Ingår i: IEEE Software. - 1937-4194 .- 0740-7459. ; 38:1, s. 123-125
-
Tidskriftsartikel (refereegranskat)abstract
- The special issue of IEEE Software, the ‘Practitioners Digest’ reports on papers about serverless application engineering from Journal of Systems and Software, the 2020 European Conference on Software Architecture, and the 19th International Conference on Middleware. ‘Function-as-a-Service Performance Evaluation: A Multivocal Literature Review’ by Joel Scheuner and Philipp Leitner describes 112 studies that report the empirical evaluation of the performance of function-as-a-service (FaaS) platforms. FaaS is a form of serverless cloud computing and is de fined as FaaS platform Web Services (AWS) executing event-triggered code snippets. This paper consolidates the results from 61 industrial and 51 academic performance studies and provides actionable recommendations on reproducible FaaS experimentation.
|
|
| 2. |
- Eismann, Simon, et al.
(författare)
-
The State of Serverless Applications: Collection, Characterization, and Community Consensus
- 2022
-
Ingår i: IEEE Transactions on Software Engineering. - 0098-5589 .- 1939-3520. ; 48:10, s. 4152-4166
-
Tidskriftsartikel (refereegranskat)abstract
- Over the last five years, all major cloud platform providers have increased their serverless offerings. Many early adopters report significant benefits for serverless-based over traditional applications, and many companies are considering moving to serverless themselves. However, currently there exist only few, scattered, and sometimes even conflicting reports on when serverless applications are well suited and what the best practices for their implementation are. We address this problem in the present study about the state of serverless applications. We collect descriptions of 89 serverless applications from open-source projects, academic literature, industrial literature, and domain-specific feedback. We analyze 16 characteristics that describe why and when successful adopters are using serverless applications, and how they are building them. We further compare the results of our characterization study to 10 existing, mostly industrial, studies and datasets; this allows us to identify points of consensus across multiple studies, investigate points of disagreement, and overall confirm the validity of our results. The results of this study can help managers to decide if they should adopt serverless technology, engineers to learn about current practices of building serverless applications, and researchers and platform providers to better understand the current landscape of serverless applications.
|
|
| 3. |
- Scheuner, Joel, 1991, et al.
(författare)
-
A Cloud Benchmark Suite Combining Micro and Applications Benchmarks
- 2018
-
Ingår i: ACM/SPEC International Conference on Performance Engineering Companion. - New York, NY, USA : ACM. - 9781450356299 ; , s. 161-166
-
Konferensbidrag (refereegranskat)abstract
- Micro and application performance benchmarks are commonly used to guide cloud service selection. However, they are often considered in isolation in a hardly reproducible setup with a flawed execution strategy. This paper presents a new execution methodology that combines micro and application benchmarks into a benchmark suite called RMIT Combined, integrates this suite into an automated cloud benchmarking environment, and implements a repeatable execution strategy. Additionally, a newly crafted Web serving benchmark called WPBench with three different load scenarios is contributed. A case study in the Amazon EC2 cloud demonstrates that choosing a cost-efficient instance type can deliver up to 40% better performance with 40% lower costs at the same time for the Web serving benchmark WPBench. Contrary to prior research, our findings reveal that network performance does not vary relevantly anymore. Our results also show that choosing a modern type of virtualization can improve disk utilization up to 10% for I/O-heavy workloads.
|
|
| 4. |
- Scheuner, Joel, 1991, et al.
(författare)
-
CrossFit: Fine-grained Benchmarking of Serverless Application Performance across Cloud Providers
- 2022
-
Ingår i: Proceedings - 2022 IEEE/ACM 15th International Conference on Utility and Cloud Computing, UCC 2022. ; , s. 51-60
-
Konferensbidrag (refereegranskat)abstract
- Serverless computing emerged as a promising cloud computing paradigm for deploying cloud-native applications but raises new performance challenges. Existing performance evaluation studies focus on micro-benchmarking to measure an individual aspect of serverless functions, such as CPU speed, but lack an in-depth analysis of differences in application performance across cloud providers. This paper presents CrossFit, an approach for detailed and fair cross-provider performance benchmarking of serverless applications based on a providerindependent tracing model. Our case study demonstrates how detailed distributed tracing enables drill-down analysis to explain performance differences between two leading cloud providers, AWS and Azure. The results for an asynchronous application show that trigger time contributes most delay to the end-to-end latency and explains the main performance difference between cloud providers. Our results further reveal how increasing and bursty workloads affect performance stability, median latency, and tail latency.
|
|
| 5. |
- Scheuner, Joel, 1991, et al.
(författare)
-
Estimating Cloud Application Performance Based on Micro-Benchmark Profiling
- 2018
-
Ingår i: 2018 IEEE 11th International Conference on Cloud Computing (CLOUD). - 2159-6190 .- 2159-6190. - 9781538672358 ; , s. 90-97
-
Konferensbidrag (refereegranskat)abstract
- The continuing growth of the cloud computing market has led to an unprecedented diversity of cloud services. To support service selection, micro-benchmarks are commonly used to identify the best performing cloud service. However, it remains unclear how relevant these synthetic micro-benchmarks are for gaining insights into the performance of real-world applications. Therefore, this paper develops a cloud benchmarking methodology that uses micro-benchmarks to profile applications and subsequently predicts how an application performs on a wide range of cloud services. A study with a real cloud provider (Amazon EC2) has been conducted to quantitatively evaluate the estimation model with 38 metrics from 23 micro-benchmarks and 2 applications from different domains. The results reveal remarkably low variability in cloud service performance and show that selected micro-benchmarks can estimate the duration of a scientific computing application with a relative error of less than 10% and the response time of a Web serving application with a relative error between 10% and 20%. In conclusion, this paper emphasizes the importance of cloud benchmarking by substantiating the suitability of micro-benchmarks for estimating application performance in comparison to common baselines but also highlights that only selected micro-benchmarks are relevant to estimate the performance of a particular application.
|
|
| 6. |
- Scheuner, Joel, 1991, et al.
(författare)
-
Function-as-a-Service Performance Evaluation: A Multivocal Literature Review
- 2020
-
Ingår i: Journal of Systems and Software. - : Elsevier BV. - 0164-1212. ; 170
-
Tidskriftsartikel (refereegranskat)abstract
- Function-as-a-Service (FaaS) is one form of the serverless cloud computing paradigm and is defined through FaaS platforms (e.g., AWS Lambda) executing event-triggered code snippets (i.e., functions). Many studies that empirically evaluate the performance of such FaaS platforms have started to appear but we are currently lacking a comprehensive understanding of the overall domain. To address this gap, we conducted a multivocal literature review (MLR) covering 112 studies from academic (51) and grey (61) literature. We find that existing work mainly studies the AWS Lambda platform and focuses on micro-benchmarks using simple functions to measure CPU speed and FaaS platform overhead (i.e., container cold starts). Further, we discover a mismatch between academic and industrial sources on tested platform configurations, find that function triggers remain insufficiently studied, and identify HTTP API gateways and cloud storages as the most used external service integrations. Following existing guidelines on experimentation in cloud systems, we discover many flaws threatening the reproducibility of experiments presented in the surveyed studies. We conclude with a discussion of gaps in literature and highlight methodological suggestions that may serve to improve future FaaS performance evaluation studies.
|
|
| 7. |
- Scheuner, Joel, 1991, et al.
(författare)
-
Performance Benchmarking of Infrastructure-as-a-Service (IaaS) Clouds with Cloud WorkBench (Tutorial)
- 2019
-
Ingår i: 2019 IEEE 4th International Workshops on Foundations and Applications of Self* Systems (FAS*W). - 9781728124063 ; June 2019, s. 257-258
-
Konferensbidrag (refereegranskat)abstract
- The continuing growth of the cloud computing market has led to an unprecedented diversity of cloud services with different performance characteristics. To support service selection, researchers and practitioners conduct cloud performance benchmarking by measuring and objectively comparing the performance of different providers and configurations (e.g., instance types in different data center regions). In this tutorial, we demonstrate how to write performance tests for IaaS clouds using the Web-based benchmarking tool Cloud WorkBench (CWB). We will motivate and introduce benchmarking of IaaS cloud in general, demonstrate the execution of a simple benchmark in a public cloud environment, summarize the CWB tool architecture, and interactively develop and deploy a more advanced benchmark together with the participants.
|
|
| 8. |
- Scheuner, Joel, 1991, et al.
(författare)
-
Performance Benchmarking of Infrastructure-as-a-Service (IaaS) Clouds with CloudWorkBench
- 2019
-
Ingår i: ICPE 2019 - Companion of the 2019 ACM/SPEC International Conference on Performance Engineering. - New York, NY, USA : ACM. ; , s. 53-56
-
Konferensbidrag (refereegranskat)abstract
- The continuing growth of the cloud computing market has led to an unprecedented diversity of cloud services with different performance characteristics. To support service selection, researchers and practitioners conduct cloud performance benchmarking by measuring and objectively comparing the performance of different providers and configurations (e.g., instance types in different data center regions). In this tutorial, we demonstrate how to write performance tests for IaaS clouds using the Web-based benchmarking tool Cloud WorkBench (CWB). We will motivate and introduce benchmarking of IaaS cloud in general, demonstrate the execution of a simple benchmark in a public cloud environment, summarize the CWB tool architecture, and interactively develop and deploy a more advanced benchmark together with the participants.
|
|
| 9. |
- Scheuner, Joel, 1991
(författare)
-
Performance Evaluation of Serverless Applications and Infrastructures
- 2022
-
Doktorsavhandling (övrigt vetenskapligt/konstnärligt)abstract
- Context. Cloud computing has become the de facto standard for deploying modern web-based software systems, which makes its performance crucial to the efficient functioning of many applications. However, the unabated growth of established cloud services, such as Infrastructure-as-a-Service (IaaS), and the emergence of new serverless services, such as Function-as-a-Service (FaaS), has led to an unprecedented diversity of cloud services with different performance characteristics. Measuring these characteristics is difficult in dynamic cloud environments due to performance variability in large-scale distributed systems with limited observability. Objective. This thesis aims to enable reproducible performance evaluation of serverless applications and their underlying cloud infrastructure. Method. A combination of literature review and empirical research established a consolidated view on serverless applications and their performance. New solutions were developed through engineering research and used to conduct performance benchmarking field experiments in cloud environments. Findings. The review of 112 FaaS performance studies from academic and industrial sources found a strong focus on a single cloud platform using artificial micro-benchmarks and discovered that most studies do not follow reproducibility principles on cloud experimentation. Characterizing 89 serverless applications revealed that they are most commonly used for short-running tasks with low data volume and bursty workloads. A novel trace-based serverless application benchmark shows that external service calls often dominate the median end-to-end latency and cause long tail latency. The latency breakdown analysis further identifies performance challenges of serverless applications, such as long delays through asynchronous function triggers, substantial runtime initialization for coldstarts, increased performance variability under bursty workloads, and heavily provider-dependent performance characteristics. The evaluation of different cloud benchmarking methodologies has shown that only selected micro-benchmarks are suitable for estimating application performance, performance variability depends on the resource type, and batch testing on the same instance with repetitions should be used for reliable performance testing. Conclusions. The insights of this thesis can guide practitioners in building performance-optimized serverless applications and researchers in reproducibly evaluating cloud performance using suitable execution methodologies and different benchmark types.
|
|
| 10. |
- Scheuner, Joel, 1991
(författare)
-
Towards Measuring and Understanding Performance in Infrastructure- and Function-as-a-Service Clouds
- 2020
-
Licentiatavhandling (övrigt vetenskapligt/konstnärligt)abstract
- Context. Cloud computing has become the de facto standard for deploying modern software systems, which makes its performance crucial to the efficient functioning of many applications. However, the unabated growth of established cloud services, such as Infrastructure-as-a-Service (IaaS), and the emergence of new services, such as Function-as-a-Service (FaaS), has led to an unprecedented diversity of cloud services with different performance characteristics. Objective. The goal of this licentiate thesis is to measure and understand performance in IaaS and FaaS clouds. My PhD thesis will extend and leverage this understanding to propose solutions for building performance-optimized FaaS cloud applications. Method. To achieve this goal, quantitative and qualitative research methods are used, including experimental research, artifact analysis, and literature review. Findings. The thesis proposes a cloud benchmarking methodology to estimate application performance in IaaS clouds, characterizes typical FaaS applications, identifies gaps in literature on FaaS performance evaluations, and examines the reproducibility of reported FaaS performance experiments. The evaluation of the benchmarking methodology yielded promising results for benchmark-based application performance estimation under selected conditions. Characterizing 89 FaaS applications revealed that they are most commonly used for short-running tasks with low data volume and bursty workloads. The review of 112 FaaS performance studies from academic and industrial sources found a strong focus on a single cloud platform using artificial micro-benchmarks and discovered that the majority of studies do not follow reproducibility principles on cloud experimentation. Future Work. Future work will propose a suite of application performance benchmarks for FaaS, which is instrumental for evaluating candidate solutions towards building performance-optimized FaaS applications.
|
|
