Lightweight Asynchronous Snapshots for Distributed Dataflows

QC 20150630
Distributed stateful stream processing enables the deployment and execution of large scale continuous computations in the cloud, targeting both low latency and high throughput. One of the most fundamental challenges of this paradigm is providing processing guarantees under potential failures. Existing approaches rely on periodic global state snapshots that can be used for failure recovery. Those approaches suffer from two main drawbacks. First, they often stall the overall computation which impacts ingestion. Second, they eagerly persist all records in transit along with the operation states which results in larger snapshots than required. In this work we propose Asynchronous Barrier Snapshotting (ABS), a lightweight algorithm suited for modern dataflow execution engines that minimises space requirements. ABS persists only operator states on acyclic execution topologies while keeping a minimal record log on cyclic dataflows. We implemented ABS on Apache Flink, a distributed analytics engine that supports stateful stream processing. Our evaluation shows that our algorithm does not have a heavy impact on the execution, maintaining linear scalability and performing well with frequent snapshots.

TEKNIK OCH TEKNOLOGIER Elektroteknik och elektronik Datorsystem hsv//swe
ENGINEERING AND TECHNOLOGY Electrical Engineering, Electronic Engineering, Information Engineering Computer Systems hsv//eng
fault tolerance
distributed computing
stream processing
dataflow
cloud computing
state management
Informations- och kommunikationsteknik
Information and Communication Technology
Computer Science
Datalogi

Fóra, GyulaCSL Computer Systems Laboratory, SICS Swedish Institute of Compute Science (author)
Ewen, StephanData Artisans GmbH (author)
Haridi, SeifKTH,Programvaruteknik och Datorsystem, SCS(Swepub:kth)u1j6y2uf (author)
Tzoumas, KostasData Artisans GmbH (author)
KTHProgramvaruteknik och Datorsystem, SCS (creator_code:org_t)

By the author/editor: Carbone, Paris, ...; Fóra, Gyula; Ewen, Stephan; Haridi, Seif; Tzoumas, Kostas

About the subject

ENGINEERING AND TECHNOLOGY: ENGINEERING AND ...; and Electrical Engin ...; and Computer Systems

Kungliga biblioteket hanterar dina personuppgifter i enlighet med EU:s dataskyddsförordning (2018), GDPR. Läs mer om hur det funkar här.
Så här hanterar KB dina uppgifter vid användning av denna tjänst.