The Comparative Evaluation of Test Prioritization Approaches in an Industrial Study

• Many test prioritisation techniques have been proposed in order to improve test effectiveness of Continuous Integration (CI) pipelines. Particularly, diversity-based testing (DBT) has shown promising and competitive results to improve test effectiveness. We report on a case study considering the CI pipeline of Axis Communications in Sweden. We compared three different prioritisation approaches (i.e., diversity, failure history and time) in terms of their impact on coverage, failure detection rates and reduction on test execution time. Our results reveal that DBT is the best candidate to provide feature coverage, whereas failure rate prioritisation yields better failure coverage. Time-based prioritisation is not a reliable approach to provide cost-effective testing. Moreover, DBT would allow stakeholders to receive quick feedback on many combinations of integrated features to verify their code changes. Our participants report that developers are mainly interested in: (i) receiving quick feedback on a high combination of integrated features to verify their code changes, and (ii) associate their test suites to confidence scores representing the risk of missing failures given that fewer tests are executed.

Ämnesord

NATURVETENSKAP  -- Data- och informationsvetenskap (hsv//swe)

NATURAL SCIENCES  -- Computer and Information Sciences (hsv//eng)

Nyckelord

comparative evaluation

continuous integration

diversity based testing

regression testing

Cost effectiveness

Failure rate

Integration testing

Code changes

Comparative evaluations

Continuous integrations

Evaluation of test

Prioritization

Prioritization techniques

Test effectiveness

Test prioritization

Pipelines

Publikations- och innehållstyp

ref (ämneskategori)

kon (ämneskategori)