onr:"swepub:oai:DiVA.org:kth-220133"
Search: onr:"swepub:oai:DiVA.org:kth-220133" > State Management in...
- 1 of 1
- Previous record
- Next record
- To hitlist
State Management in Apache Flink : Consistent Stateful Distributed Stream Processing
-
- Carbone, Paris (author)
- KTH,Programvaruteknik och Datorsystem, SCS
-
- Ewen, Stephan (author)
- data Artisans
-
- Fóra, Gyula (author)
- King Digital Entertainment Limited
- show more...
- show less...
- (creator_code:org_t)
- 2017-08
- 2017
- English.
- In: Proceedings of the VLDB Endowment. - : ACM Digital Library. - 2150-8097. ; 10, s. 1718-1729
- Journal article (peer-reviewed)
Abstract
Subject headings
Close
- Stream processors are emerging in industry as an apparatus that drives analytical but also mission critical services handling the core of persistent application logic. Thus, apart from scalability and low-latency, a rising system need is first-class support for application state together with strong consistency guarantees, and adaptivity to cluster reconfigurations, software patches and partial failures. Although prior systems research has addressed some of these specific problems, the practical challenge lies on how such guarantees can be materialized in a transparent, non-intrusive manner that relieves the user from unnecessary constraints. Such needs served as the main design principles of state management in Apache Flink, an open source, scalable stream processor.We present Flink’s core pipelined, in-flight mechanism which guarantees the creation of lightweight, consistent, distributed snapshots of application state, progressively, without impacting continuous execution. Consistent snapshots cover all needs for system reconfiguration, fault tolerance and version management through coarse grained rollback recovery. Application state is declared explicitly to the system, allowing efficient partitioning and transparent commits to persistent storage. We further present Flink’s backend implementations and mechanisms for high availability, external state queries and output commit. Finally, we demonstrate how these mechanisms behave in practice with metrics and large deployment insights exhibiting the low performance trade-offs of our approach and the general benefits of exploiting asynchrony in continuous, yet sustainable system deployments.
Subject headings
- TEKNIK OCH TEKNOLOGIER -- Elektroteknik och elektronik -- Datorsystem (hsv//swe)
- ENGINEERING AND TECHNOLOGY -- Electrical Engineering, Electronic Engineering, Information Engineering -- Computer Systems (hsv//eng)
Keyword
- stream processing
- data management
- computer systems
- databases
- data processing
- data science
- big data
- Datalogi
- Computer Science
Publication and Content Type
- ref (subject category)
- art (subject category)
Find in a library
- Proceedings of the VLDB Endowment (Search for host publication in LIBRIS)
To the university's database
- 1 of 1
- Previous record
- Next record
- To hitlist
Find more in SwePub
- By the author/editor
- Carbone, Paris
- Ewen, Stephan
- Fóra, Gyula
- Haridi, Seif, 19 ...
- Richter, Stefan
- Tzoumas, Kostas
- About the subject
-
- ENGINEERING AND TECHNOLOGY
- ENGINEERING AND ...
- and Electrical Engin ...
- and Computer Systems
- Articles in the publication
- Proceedings of t ...
- By the university
- Royal Institute of Technology
Search outside SwePub
- Extend your search to:
- Google Book Search
- Google Scholar
Kungliga biblioteket hanterar dina personuppgifter i enlighet med EU:s dataskyddsförordning (2018), GDPR. Läs mer om hur det funkar här.
Så här hanterar KB dina uppgifter vid användning av denna tjänst.
- Copyright © LIBRIS - National Library Systems
- LIBRIS.kb.se
