| 2. |
- Gutierrez, Felipe, et al.
(författare)
-
AdCom : Adaptive combiner for streaming aggregations
- 2021
-
Ingår i: Advances in Database Technology - EDBT. - : OpenProceedings. - 9783893180844 ; , s. 403-414
-
Konferensbidrag (refereegranskat)abstract
- Continuous applications such as device monitoring and anomaly detection often require real-time aggregated statistics over unbounded data streams. While existing stream processing systems such as Flink, Spark, and Storm support processing of streaming aggregations, their optimizations are limited with respect to the dynamic nature of the data, and therefore are suboptimal when the workload changes and/or when there is data skew. In this paper we present AdCom, which is an adaptive combiner for stream processing engines. The use of AdCom in aggregation queries enables pre-aggregating tuples upstream (i.e., before data shuffling) followed by global aggregation downstream. In contrast to existing approaches, AdCom can automatically adjust the number of tuples to pre-aggregate depending on the data rate and available network. Our experimental study using real-world streaming workloads shows that using AdCom leads to 2.5-9× higher sustainable throughput without compromising latency.
|
|
| 3. |
- Kalavri, Vasiliki, et al.
(författare)
-
Asymmetry in Large-Scale Graph Analysis, Explained
- 2014
-
Ingår i: <em>Proceedings of the Second International Workshop on Graph Data ManagementExperience and Systems (GRADES 2014)</em>, June 22, 2014, Snowbird, Utah, USA.. - New York, NY, USA : ACM.
-
Konferensbidrag (refereegranskat)abstract
- Iterative computations are in the core of large-scale graph processing. In these applications, a set of parameters is continuously refined, until a fixed point is reached. Such fixed point iterations often exhibit non-uniform computational behavior, where changes propagate with different speeds throughout the parameter set, making them active or inactive during iterations. This asymmetrical behavior can lead to a many redundant computations, if not exploited. Many specialized graph processing systems and APIs exist that run iterative algorithms efficiently exploiting this asymmetry. However, their functionality is sometimes vaguely defined and due to their different programming models and terminology used, it is often challenging to derive equivalence between them. We describe an optimization framework for iterative graph processing, which utilizes dataset dependencies. We explain several optimization techniques that exploit asymmetrical behavior of graph algorithms. We formally specify the conditions under which, an algorithm can use a certain technique. We also design template execution plans, using a canonical set of dataflow operators and we evaluate them using real-world datasets and applications. Our experiments show that optimized plans can significantly reduce execution time, often by an order of magnitude. Based on our experiments, we identify a trade-off that can be easily captured and could serve as the basis for automatic optimization of large-scale graph-processing applications.
|
|
