A Multi-factor Approach for Flaky Test Detection and Automated Root Cause Analysis

↓ Direkt till sidans innehåll
↓ Direkt till sidans sekundära innehåll (sidomenyn)

Search: WFRF:(Leifler Ola) > A Multi-factor Appr...

Details
MARC

Ahmad, AzeemLinköpings universitet,Programvara och system,Tekniska fakulteten (author)

A Multi-factor Approach for Flaky Test Detection and Automated Root Cause Analysis

Article/chapterEnglish2021

Publisher, publication year, extent ...

IEEE COMPUTER SOC,2021

Numbers

LIBRIS-ID:oai:gup.ub.gu.se/316073
https://gup.ub.gu.se/publication/316073URI
https://doi.org/10.1109/APSEC53868.2021.00041DOI
https://urn.kb.se/resolve?urn=urn:nbn:se:liu:diva-186181URI

Supplementary language notes

Language:English

Part of subdatabase

SwePubSwePub

Classification

Subject category:ref swepub-contenttype
Subject category:kon swepub-publicationtype

Notes

Developers often spend time to determine whether test case failures are real failures or flaky. The flaky tests, also known as non-deterministic tests, switch their outcomes without any modification in the codebase, hence reducing the confidence of developers during maintenance as well as in the quality of a product. Re-running test cases to reveal flakiness is resource-consuming, unreliable and does not reveal the root causes of test flakiness. Our paper evaluates a multi-factor approach to identify flaky test executions implemented in a tool named MDF laker. The four factors are: trace-back coverage, flaky frequency, number of test smells, and test size. Based on the extracted factors, MDFlaker uses k-Nearest Neighbor (KNN) to determine whether failed test executions are flaky. We investigate MDFlaker in a case study with 2166 test executions from different open-source repositories. We evaluate the effectiveness of our flaky detection tool. We illustrate how the multi-factor approach can be used to reveal root causes for flakiness, and we conduct a qualitative comparison between MDF laker and other tools proposed in literature. Our results show that the combination of different factors can be used to identify flaky tests. Each factor has its own trade-off, e.g., trace-back leads to many true positives, while flaky frequency yields more true negatives. Therefore, specific combinations of factors enable classification for testers with limited information (e.g., not enough test history information).

Subject headings and genre

NATURVETENSKAP Data- och informationsvetenskap Programvaruteknik hsv//swe
NATURAL SCIENCES Computer and Information Sciences Software Engineering hsv//eng
NATURVETENSKAP Data- och informationsvetenskap Datavetenskap hsv//swe
NATURAL SCIENCES Computer and Information Sciences Computer Sciences hsv//eng
automated root-cause analysis
flaky test detection
flaky tests
non-deterministic tests
trace-back
flaky tests; non-deterministic tests; flaky test detection; automated root-cause analysis; trace-back

Added entries (persons, corporate bodies, meetings, titles ...)

de Oliveira Neto, Francisco GomesGothenburg University,Göteborgs universitet,Institutionen för data- och informationsteknik (GU),Department of Computer Science and Engineering (GU),Chalmers & Univ Gothenburg, Sweden(Swepub:gu)xdeofr (author)
Shi, ZhixiangLinköpings universitet,Institutionen för datavetenskap,Tekniska fakulteten(Swepub:liu)n/a (author)
Sandahl, KristianLinköpings universitet,Programvara och system,Tekniska fakulteten(Swepub:liu)krisa34 (author)
Leifler, OlaLinköpings universitet,Programvara och system,Tekniska fakulteten(Swepub:liu)olale55 (author)
Linköpings universitetProgramvara och system (creator_code:org_t)

Related titles

In:Proceedings - Asia-Pacific Software Engineering Conference, APSEC: IEEE COMPUTER SOC, s. 338-3481530-1362
In:2021 28TH ASIA-PACIFIC SOFTWARE ENGINEERING CONFERENCE (APSEC 2021): IEEE COMPUTER SOC, s. 338-34897816654378449781665437851

Internet link

Find in a library

Proceedings - Asia-Pacific Software Engineering Conference, APSEC (Search for host publication in LIBRIS)

To the university's database

Find more in SwePub

By the author/editor: Ahmad, Azeem; de Oliveira Neto ...; Shi, Zhixiang; Sandahl, Kristia ...; Leifler, Ola

About the subject

NATURAL SCIENCES: NATURAL SCIENCES; and Computer and Inf ...; and Software Enginee ...

NATURAL SCIENCES: NATURAL SCIENCES; and Computer and Inf ...; and Computer Science ...

Articles in the publication: Proceedings - As ...; 2021 28TH ASIA-P ...

By the university: University of Gothenburg; Linköping University

Search outside SwePub

Extend your search to:: Google; Google Book Search; Google Scholar

Kungliga biblioteket hanterar dina personuppgifter i enlighet med EU:s dataskyddsförordning (2018), GDPR. Läs mer om hur det funkar här.
Så här hanterar KB dina uppgifter vid användning av denna tjänst.

LIBRIS.kb.se