1. |
- Carver, Jeffrey C., et al.
(author)
-
(Research) Insights for Serverless Application Engineering
- 2021
-
In: IEEE Software. - 1937-4194 .- 0740-7459. ; 38:1, s. 123-125
-
Journal article (peer-reviewed)abstract
- The special issue of IEEE Software, the ‘Practitioners Digest’ reports on papers about serverless application engineering from Journal of Systems and Software, the 2020 European Conference on Software Architecture, and the 19th International Conference on Middleware. ‘Function-as-a-Service Performance Evaluation: A Multivocal Literature Review’ by Joel Scheuner and Philipp Leitner describes 112 studies that report the empirical evaluation of the performance of function-as-a-service (FaaS) platforms. FaaS is a form of serverless cloud computing and is de fined as FaaS platform Web Services (AWS) executing event-triggered code snippets. This paper consolidates the results from 61 industrial and 51 academic performance studies and provides actionable recommendations on reproducible FaaS experimentation.
|
|
2. |
- Eismann, Simon, et al.
(author)
-
The State of Serverless Applications: Collection, Characterization, and Community Consensus
- 2022
-
In: IEEE Transactions on Software Engineering. - 0098-5589 .- 1939-3520. ; 48:10, s. 4152-4166
-
Journal article (peer-reviewed)abstract
- Over the last five years, all major cloud platform providers have increased their serverless offerings. Many early adopters report significant benefits for serverless-based over traditional applications, and many companies are considering moving to serverless themselves. However, currently there exist only few, scattered, and sometimes even conflicting reports on when serverless applications are well suited and what the best practices for their implementation are. We address this problem in the present study about the state of serverless applications. We collect descriptions of 89 serverless applications from open-source projects, academic literature, industrial literature, and domain-specific feedback. We analyze 16 characteristics that describe why and when successful adopters are using serverless applications, and how they are building them. We further compare the results of our characterization study to 10 existing, mostly industrial, studies and datasets; this allows us to identify points of consensus across multiple studies, investigate points of disagreement, and overall confirm the validity of our results. The results of this study can help managers to decide if they should adopt serverless technology, engineers to learn about current practices of building serverless applications, and researchers and platform providers to better understand the current landscape of serverless applications.
|
|
3. |
- Scheuner, Joel, 1991, et al.
(author)
-
A Cloud Benchmark Suite Combining Micro and Applications Benchmarks
- 2018
-
In: ACM/SPEC International Conference on Performance Engineering Companion. - New York, NY, USA : ACM. - 9781450356299 ; , s. 161-166
-
Conference paper (peer-reviewed)abstract
- Micro and application performance benchmarks are commonly used to guide cloud service selection. However, they are often considered in isolation in a hardly reproducible setup with a flawed execution strategy. This paper presents a new execution methodology that combines micro and application benchmarks into a benchmark suite called RMIT Combined, integrates this suite into an automated cloud benchmarking environment, and implements a repeatable execution strategy. Additionally, a newly crafted Web serving benchmark called WPBench with three different load scenarios is contributed. A case study in the Amazon EC2 cloud demonstrates that choosing a cost-efficient instance type can deliver up to 40% better performance with 40% lower costs at the same time for the Web serving benchmark WPBench. Contrary to prior research, our findings reveal that network performance does not vary relevantly anymore. Our results also show that choosing a modern type of virtualization can improve disk utilization up to 10% for I/O-heavy workloads.
|
|
4. |
- Scheuner, Joel, 1991, et al.
(author)
-
CrossFit: Fine-grained Benchmarking of Serverless Application Performance across Cloud Providers
- 2022
-
In: Proceedings - 2022 IEEE/ACM 15th International Conference on Utility and Cloud Computing, UCC 2022. ; , s. 51-60
-
Conference paper (peer-reviewed)abstract
- Serverless computing emerged as a promising cloud computing paradigm for deploying cloud-native applications but raises new performance challenges. Existing performance evaluation studies focus on micro-benchmarking to measure an individual aspect of serverless functions, such as CPU speed, but lack an in-depth analysis of differences in application performance across cloud providers. This paper presents CrossFit, an approach for detailed and fair cross-provider performance benchmarking of serverless applications based on a providerindependent tracing model. Our case study demonstrates how detailed distributed tracing enables drill-down analysis to explain performance differences between two leading cloud providers, AWS and Azure. The results for an asynchronous application show that trigger time contributes most delay to the end-to-end latency and explains the main performance difference between cloud providers. Our results further reveal how increasing and bursty workloads affect performance stability, median latency, and tail latency.
|
|
5. |
- Scheuner, Joel, 1991, et al.
(author)
-
Estimating Cloud Application Performance Based on Micro-Benchmark Profiling
- 2018
-
In: 2018 IEEE 11th International Conference on Cloud Computing (CLOUD). - 2159-6190 .- 2159-6190. - 9781538672358 ; 2018-July, s. 90-97
-
Conference paper (peer-reviewed)abstract
- The continuing growth of the cloud computing market has led to an unprecedented diversity of cloud services. To support service selection, micro-benchmarks are commonly used to identify the best performing cloud service. However, it remains unclear how relevant these synthetic micro-benchmarks are for gaining insights into the performance of real-world applications. Therefore, this paper develops a cloud benchmarking methodology that uses micro-benchmarks to profile applications and subsequently predicts how an application performs on a wide range of cloud services. A study with a real cloud provider (Amazon EC2) has been conducted to quantitatively evaluate the estimation model with 38 metrics from 23 micro-benchmarks and 2 applications from different domains. The results reveal remarkably low variability in cloud service performance and show that selected micro-benchmarks can estimate the duration of a scientific computing application with a relative error of less than 10% and the response time of a Web serving application with a relative error between 10% and 20%. In conclusion, this paper emphasizes the importance of cloud benchmarking by substantiating the suitability of micro-benchmarks for estimating application performance in comparison to common baselines but also highlights that only selected micro-benchmarks are relevant to estimate the performance of a particular application.
|
|
6. |
- Scheuner, Joel, 1991, et al.
(author)
-
Function-as-a-Service Performance Evaluation: A Multivocal Literature Review
- 2020
-
In: Journal of Systems and Software. - : Elsevier BV. - 0164-1212. ; 170
-
Journal article (peer-reviewed)abstract
- Function-as-a-Service (FaaS) is one form of the serverless cloud computing paradigm and is defined through FaaS platforms (e.g., AWS Lambda) executing event-triggered code snippets (i.e., functions). Many studies that empirically evaluate the performance of such FaaS platforms have started to appear but we are currently lacking a comprehensive understanding of the overall domain. To address this gap, we conducted a multivocal literature review (MLR) covering 112 studies from academic (51) and grey (61) literature. We find that existing work mainly studies the AWS Lambda platform and focuses on micro-benchmarks using simple functions to measure CPU speed and FaaS platform overhead (i.e., container cold starts). Further, we discover a mismatch between academic and industrial sources on tested platform configurations, find that function triggers remain insufficiently studied, and identify HTTP API gateways and cloud storages as the most used external service integrations. Following existing guidelines on experimentation in cloud systems, we discover many flaws threatening the reproducibility of experiments presented in the surveyed studies. We conclude with a discussion of gaps in literature and highlight methodological suggestions that may serve to improve future FaaS performance evaluation studies.
|
|
7. |
- Scheuner, Joel, 1991, et al.
(author)
-
Performance Benchmarking of Infrastructure-as-a-Service (IaaS) Clouds with Cloud WorkBench (Tutorial)
- 2019
-
In: 2019 IEEE 4th International Workshops on Foundations and Applications of Self* Systems (FAS*W). - 9781728124063 ; June 2019, s. 257-258
-
Conference paper (peer-reviewed)abstract
- The continuing growth of the cloud computing market has led to an unprecedented diversity of cloud services with different performance characteristics. To support service selection, researchers and practitioners conduct cloud performance benchmarking by measuring and objectively comparing the performance of different providers and configurations (e.g., instance types in different data center regions). In this tutorial, we demonstrate how to write performance tests for IaaS clouds using the Web-based benchmarking tool Cloud WorkBench (CWB). We will motivate and introduce benchmarking of IaaS cloud in general, demonstrate the execution of a simple benchmark in a public cloud environment, summarize the CWB tool architecture, and interactively develop and deploy a more advanced benchmark together with the participants.
|
|
8. |
- Scheuner, Joel, 1991, et al.
(author)
-
Performance Benchmarking of Infrastructure-as-a-Service (IaaS) Clouds with CloudWorkBench
- 2019
-
In: ICPE 2019 - Companion of the 2019 ACM/SPEC International Conference on Performance Engineering. - New York, NY, USA : ACM. ; , s. 53-56
-
Conference paper (peer-reviewed)abstract
- The continuing growth of the cloud computing market has led to an unprecedented diversity of cloud services with different performance characteristics. To support service selection, researchers and practitioners conduct cloud performance benchmarking by measuring and objectively comparing the performance of different providers and configurations (e.g., instance types in different data center regions). In this tutorial, we demonstrate how to write performance tests for IaaS clouds using the Web-based benchmarking tool Cloud WorkBench (CWB). We will motivate and introduce benchmarking of IaaS cloud in general, demonstrate the execution of a simple benchmark in a public cloud environment, summarize the CWB tool architecture, and interactively develop and deploy a more advanced benchmark together with the participants.
|
|
9. |
- Scheuner, Joel, 1991
(author)
-
Performance Evaluation of Serverless Applications and Infrastructures
- 2022
-
Doctoral thesis (other academic/artistic)abstract
- Context. Cloud computing has become the de facto standard for deploying modern web-based software systems, which makes its performance crucial to the efficient functioning of many applications. However, the unabated growth of established cloud services, such as Infrastructure-as-a-Service (IaaS), and the emergence of new serverless services, such as Function-as-a-Service (FaaS), has led to an unprecedented diversity of cloud services with different performance characteristics. Measuring these characteristics is difficult in dynamic cloud environments due to performance variability in large-scale distributed systems with limited observability. Objective. This thesis aims to enable reproducible performance evaluation of serverless applications and their underlying cloud infrastructure. Method. A combination of literature review and empirical research established a consolidated view on serverless applications and their performance. New solutions were developed through engineering research and used to conduct performance benchmarking field experiments in cloud environments. Findings. The review of 112 FaaS performance studies from academic and industrial sources found a strong focus on a single cloud platform using artificial micro-benchmarks and discovered that most studies do not follow reproducibility principles on cloud experimentation. Characterizing 89 serverless applications revealed that they are most commonly used for short-running tasks with low data volume and bursty workloads. A novel trace-based serverless application benchmark shows that external service calls often dominate the median end-to-end latency and cause long tail latency. The latency breakdown analysis further identifies performance challenges of serverless applications, such as long delays through asynchronous function triggers, substantial runtime initialization for coldstarts, increased performance variability under bursty workloads, and heavily provider-dependent performance characteristics. The evaluation of different cloud benchmarking methodologies has shown that only selected micro-benchmarks are suitable for estimating application performance, performance variability depends on the resource type, and batch testing on the same instance with repetitions should be used for reliable performance testing. Conclusions. The insights of this thesis can guide practitioners in building performance-optimized serverless applications and researchers in reproducibly evaluating cloud performance using suitable execution methodologies and different benchmark types.
|
|
10. |
- Scheuner, Joel, 1991
(author)
-
Towards Measuring and Understanding Performance in Infrastructure- and Function-as-a-Service Clouds
- 2020
-
Licentiate thesis (other academic/artistic)abstract
- Context. Cloud computing has become the de facto standard for deploying modern software systems, which makes its performance crucial to the efficient functioning of many applications. However, the unabated growth of established cloud services, such as Infrastructure-as-a-Service (IaaS), and the emergence of new services, such as Function-as-a-Service (FaaS), has led to an unprecedented diversity of cloud services with different performance characteristics. Objective. The goal of this licentiate thesis is to measure and understand performance in IaaS and FaaS clouds. My PhD thesis will extend and leverage this understanding to propose solutions for building performance-optimized FaaS cloud applications. Method. To achieve this goal, quantitative and qualitative research methods are used, including experimental research, artifact analysis, and literature review. Findings. The thesis proposes a cloud benchmarking methodology to estimate application performance in IaaS clouds, characterizes typical FaaS applications, identifies gaps in literature on FaaS performance evaluations, and examines the reproducibility of reported FaaS performance experiments. The evaluation of the benchmarking methodology yielded promising results for benchmark-based application performance estimation under selected conditions. Characterizing 89 FaaS applications revealed that they are most commonly used for short-running tasks with low data volume and bursty workloads. The review of 112 FaaS performance studies from academic and industrial sources found a strong focus on a single cloud platform using artificial micro-benchmarks and discovered that the majority of studies do not follow reproducibility principles on cloud experimentation. Future Work. Future work will propose a suite of application performance benchmarks for FaaS, which is instrumental for evaluating candidate solutions towards building performance-optimized FaaS applications.
|
|
11. |
- Scheuner, Joel, 1991, et al.
(author)
-
Transpiling Applications into Optimized Serverless Orchestrations
- 2019
-
In: Proceedings - 2019 IEEE 4th International Workshops on Foundations and Applications of Self* Systems, FAS*W 2019. - : IEEE. ; June 2019, s. 72-73
-
Conference paper (peer-reviewed)abstract
- The serverless computing paradigm promises increased development productivity by abstracting the underlying hardware infrastructure and software runtime when building distributed cloud applications. However, composing a serverless application consisting of many tiny functions is still a cumbersome and inflexible process due to the lack of a unified source code view and strong coupling to non-standardized function-level interfaces for code and configuration. In our vision, developers can focus on writing readable source code in a logical structure, which then gets transformed into an optimized multi-function serverless orchestration. Our idea involves transpilation (i.e., source-to-source transformation) based on an optimization model (e.g., cost optimization) by dynamically deciding which set of methods will be grouped into individual deployment units. A successful implementation of our vision would enable a broader range of serverless applications and allow for dynamic deployment optimization based on monitoring runtime metrics. Further, we would expect increased developer productivity by using more familiar abstractions and facilitating clean coding practices and code reuse.
|
|
12. |
- Scheuner, Joel, 1991, et al.
(author)
-
TriggerBench: A Performance Benchmark for Serverless Function Triggers
- 2022
-
In: Proceedings - 2022 IEEE International Conference on Cloud Engineering, IC2E 2022. ; , s. 96-103
-
Conference paper (peer-reviewed)abstract
- Serverless computing offers a scalable event-based paradigm for deploying managed cloud-native applications. Function triggers are essential building blocks in serverless, as they initiate any function execution. However, function triggering is insufficiently studied and inherently hard to measure given the distributed, ephemeral, and asynchronous nature of event-based function coordination. To address this gap, we present TriggerBench, a cross-provider benchmark for evaluating serverless function triggers based on distributed tracing. We evaluate the trigger latency (i.e., time to transition between two functions) of eight types of triggers in Microsoft Azure and three in AWS. Our results show that all triggers suffer from long tail latency, storage triggers introduce variable multi-second delays, and HTTP triggers are most suitable for interactive applications. Our insights can guide developers in choosing optimal event or messaging triggers for latency-sensitive applications. Researchers can extend TriggerBench to study the latency, scalability, and reliability of further trigger types and cloud providers.
|
|
13. |
- Schirmer, Trever, et al.
(author)
-
Fusionize: Improving Serverless Application Performance through Feedback-Driven Function Fusion
- 2022
-
In: Proceedings - 2022 IEEE International Conference on Cloud Engineering, IC2E 2022. ; , s. 85-95
-
Conference paper (peer-reviewed)abstract
- Serverless computing increases developer productivity by removing operational concerns such as managing hardware or software runtimes. Developers, however, still need to partition their application into functions, which can be error-prone and adds complexity: Using a small function size where only the smallest logical unit of an application is inside a function maximizes flexibility and reusability. Yet, having small functions leads to invocation overheads, additional cold starts, and may increase cost due to double billing during synchronous invocations. In this paper we present Fusionize, a framework that removes these concerns from developers by automatically fusing the application code into a multi-function orchestration with varying function size. Developers only need to write the application code following a lightweight programming model and do not need to worry how the application is turned into functions. Our framework automatically fuses different parts of the application into functions and manages their interactions. Leveraging monitoring data, the framework optimizes the distribution of application parts to functions to optimize deployment goals such as end-to-end latency and cost. Using two example applications, we show that Fusionizecan automatically and iteratively improve the deployment artifacts of the application.
|
|
14. |
- van Eyk, Erwin, et al.
(author)
-
Beyond Microbenchmarks: The SPEC-RG Vision for a Comprehensive Serverless Benchmark
- 2020
-
In: Companion of the ACM/SPEC International Conference on Performance Engineering. - New York, NY, USA : ACM. - 9781450371094 ; , s. 26-31
-
Conference paper (peer-reviewed)abstract
- Serverless computing services, such as Function-as-a-Service (FaaS), hold the attractive promise of a high level of abstraction and high performance, combined with the minimization of operational logic. Several large ecosystems of serverless platforms, both open- and closed-source, aim to realize this promise. Consequently, a lucrative market has emerged. However, the performance trade-offs of these systems are not well-understood. Moreover, it is exactly the high level of abstraction and the opaqueness of the operational-side that make performance evaluation studies of serverless platforms challenging. Learning from the history of IT platforms, we argue that a benchmark for serverless platforms could help address this challenge. We envision a comprehensive serverless benchmark, which we contrast to the narrow focus of prior work in this area. We argue that a comprehensive benchmark will need to take into account more than just runtime overhead, and include notions of cost, realistic workloads, more (open-source) platforms, and cloud integrations. Finally, we show through preliminary real-world experiments how such a benchmark can help compare the performance overhead when running a serverless workload on state-of-the-art platforms.
|
|