A survey of big data pipeline orchestration tools from the perspective of the datacloud project

• This paper presents a survey of existing tools for Big Data pipeline orchestration based on a comparative framework developed in the DataCloud project. We propose criteria for evaluating the tools to support reusability, flexible pipeline communication modes, and separation of concerns in Big Data pipeline descriptions. This survey aims to identify research and technological gaps and to recommend approaches for filling them. Further work in the DataCloud project is oriented towards the design, implementation, and practical evaluation of the recommended approaches. 

Subject headings

NATURVETENSKAP  -- Data- och informationsvetenskap -- Datavetenskap (hsv//swe)

NATURAL SCIENCES  -- Computer and Information Sciences -- Computer Sciences (hsv//eng)

Keyword

Big data pipeline

Orchestration tools

Reusability

Big data

Pipelines

Surveys

Communication mode

Data pipelines

Design implementation

Flexible pipeline

Further works

Orchestration tool

Pipeline communication

Separation of concerns

Publication and Content Type

ref (subject category)

kon (subject category)