Towards automated detection of data pipeline faults

Munappy, Aiswarya Raj,1990Chalmers tekniska högskola,Chalmers University of Technology,Chalmers (författare)

Data pipelines play an important role throughout the data management process. It automates the steps ranging from data generation to data reception thereby reducing the human intervention. A failure or fault in a single step of a data pipeline has cascading effects that might result in hours of manual intervention and clean-up. Data pipeline failure due to faults at different stages of data pipelines is a common challenge that eventually leads to significant performance degradation of data-intensive systems. To ensure early detection of these faults and to increase the quality of the data products, continuous monitoring and fault detection mechanism should be included in the data pipeline. In this study, we have explored the need for incorporating automated fault detection mechanisms and mitigation strategies at different stages of the data pipeline. Further, we identified faults at different stages of the data pipeline and possible mitigation strategies that can be adopted for reducing the impact of data pipeline faults thereby improving the quality of data products. The idea of incorporating fault detection and mitigation strategies is validated by realizing a small part of the data pipeline using action research in the analytics team at a large software-intensive organization within the telecommunication domain.

Bosch, Jan,1967Chalmers tekniska högskola,Chalmers University of Technology,Chalmers(Swepub:cth)janbo (författare)
Olsson, Helena HolmströmMalmö universitet,Institutionen för datavetenskap och medieteknik (DVMT)(Swepub:mau)ab2385 (författare)
Wang, Tian J.Telefonaktiebolaget L M Ericsson,Ericsson,Ericsson, Gothenburg, Sweden. (författare)
Chalmers tekniska högskolaChalmers (creator_code:org_t)

Ingår i:Proceedings - Asia-Pacific Software Engineering Conference, APSEC: IEEE2020-December, s. 346-355, s. 346-3551530-1362
Ingår i:2020 27TH ASIA-PACIFIC SOFTWARE ENGINEERING CONFERENCE (APSEC 2020): IEEE2020-December, s. 346-355, s. 346-3559781728195537

Av författaren/redakt...: Munappy, Aiswary ...; Bosch, Jan, 1967; Olsson, Helena H ...; Wang, Tian J.

Om ämnet

NATURVETENSKAP: NATURVETENSKAP; och Data och informa ...; och Annan data och i ...

NATURVETENSKAP: NATURVETENSKAP; och Data och informa ...; och Programvarutekni ...

Kungliga biblioteket hanterar dina personuppgifter i enlighet med EU:s dataskyddsförordning (2018), GDPR. Läs mer om hur det funkar här.
Så här hanterar KB dina uppgifter vid användning av denna tjänst.