Test Maintenance for Machine Learning Systems: A Case Study in the Automotive Industry

• Machine Learning (ML) systems have seen widespread use for automated decision making. Testing is essential to ensure the quality of these systems, especially safety-critical autonomous systems in the automotive domain. ML systems introduce new challenges with the potential to affect test maintenance, the process of updating test cases to match the evolving system. We conducted an exploratory case study in the automotive domain to identify factors that affect test maintenance for ML systems, as well as to make recommendations to improve the maintenance process. Based on interview and artifact analysis, we identified 14 factors affecting maintenance, including five especially relevant for ML systems—with the most important relating to non-determinism and large input spaces. We also proposed ten recommendations for improving test maintenance, including four targeting ML systems—in particular, emphasizing the use of test oracles tolerant to acceptable non-determinism. The study’s findings expand our knowledge of test maintenance for an emerging class of systems, benefiting the practitioners testing these systems.

Ämnesord

TEKNIK OCH TEKNOLOGIER  -- Maskinteknik -- Produktionsteknik, arbetsvetenskap och ergonomi (hsv//swe)

ENGINEERING AND TECHNOLOGY  -- Mechanical Engineering -- Production Engineering, Human Work Science and Ergonomics (hsv//eng)

NATURVETENSKAP  -- Data- och informationsvetenskap -- Programvaruteknik (hsv//swe)

NATURAL SCIENCES  -- Computer and Information Sciences -- Software Engineering (hsv//eng)

TEKNIK OCH TEKNOLOGIER  -- Elektroteknik och elektronik -- Datorsystem (hsv//swe)

ENGINEERING AND TECHNOLOGY  -- Electrical Engineering, Electronic Engineering, Information Engineering -- Computer Systems (hsv//eng)

Nyckelord

Software Testing

Automotive Software

Test Evolution

Test Maintenance

Machine Learning

Software Testing

Test Maintenance

Test Evolution

Machine Learning

Automotive Software

Publikations- och innehållstyp

kon (ämneskategori)

ref (ämneskategori)