Sökning: id:"swepub:oai:DiVA.org:kth-176121" >
The power of both c...
The power of both choices : Practical load balancing for distributed stream processing engines
-
- Nasir, M. Anis U. (författare)
- KTH,Kommunikationsnät
-
De Francisci Morales, G. (författare)
-
García-Soriano, D. (författare)
-
visa fler...
-
Kourtellis, N. (författare)
-
Serafini, M. (författare)
-
visa färre...
-
(creator_code:org_t)
- IEEE conference proceedings, 2015
- 2015
- Engelska.
-
Ingår i: Proceedings - International Conference on Data Engineering. - : IEEE conference proceedings. - 9781479979639 ; , s. 137-148
- Relaterad länk:
-
https://urn.kb.se/re...
-
visa fler...
-
https://doi.org/10.1...
-
visa färre...
Abstract
Ämnesord
Stäng
- We study the problem of load balancing in distributed stream processing engines, which is exacerbated in the presence of skew. We introduce Partial Key Grouping (PKG), a new stream partitioning scheme that adapts the classical 'power of two choices' to a distributed streaming setting by leveraging two novel techniques: key splitting and local load estimation. In so doing, it achieves better load balancing than key grouping while being more scalable than shuffle grouping. We test PKG on several large datasets, both real-world and synthetic. Compared to standard hashing, PKG reduces the load imbalance by up to several orders of magnitude, and often achieves nearly-perfect load balance. This result translates into an improvement of up to 60% in throughput and up to 45% in latency when deployed on a real Storm cluster.
Ämnesord
- TEKNIK OCH TEKNOLOGIER -- Elektroteknik och elektronik (hsv//swe)
- ENGINEERING AND TECHNOLOGY -- Electrical Engineering, Electronic Engineering, Information Engineering (hsv//eng)
Nyckelord
- Balancing
- Distributed parameter control systems
- Engines
- Distributed stream processing
- Distributed streaming
- Large datasets
- Load balance
- Load imbalance
- Novel techniques
- Orders of magnitude
- Power-of-two
- Network management
Publikations- och innehållstyp
- ref (ämneskategori)
- kon (ämneskategori)
Hitta via bibliotek
Till lärosätets databas