Size matters? Or not: A/B testing with limited sample in automotive embedded software

Liu, Yuchu,1992Volvo Cars, Gothenburg, Sweden.,Chalmers tekniska högskola,Chalmers University of Technology (författare)

Size matters? Or not : A/B testing with limited sample in automotive embedded software

A/B testing is gaining attention in the automotive sector as a promising tool to measure casual effects from software changes. Different from the web-facing businesses, where A/B testing has been well-established, the automotive domain often suffers from limited eligible users to participate in online experiments. To address this shortcoming, we present a method for designing balanced control and treatment groups so that sound conclusions can be drawn from experiments with considerably small sample sizes. While the Balance Match Weighted method has been used in other domains such as medicine, this is the first paper to apply and evaluate it in the context of software development. Furthermore, we describe the Balance Match Weighted method in detail and we conduct a case study together with an automotive manufacturer to apply the group design method in a fleet of vehicles. Finally, we present our case study in the automotive software engineering domain, as well as a discussion on the benefits and limitations of the A/B group design method.

Issa Mattos, David,1990Chalmers Univ Technol, Comp Sci & Engn, Gothenburg, Sweden.,Chalmers tekniska högskola,Chalmers University of Technology(Swepub:cth)davidis (författare)
Bosch, Jan,1967Chalmers Univ Technol, Comp Sci & Engn, Gothenburg, Sweden.,Chalmers tekniska högskola,Chalmers University of Technology(Swepub:cth)janbo (författare)
Olsson, Helena HolmströmMalmö universitet,Institutionen för datavetenskap och medieteknik (DVMT),Malmö Univ, Comp Sci & Media Technol, Malmö, Sweden.,Malmö university(Swepub:mau)ab2385 (författare)
Lantz, JonnVolvo Cars, Gothenburg, Sweden. (författare)
Volvo Cars, Gothenburg, Sweden.Volvo Cars (creator_code:org_t)

Ingår i:2021 47TH EUROMICRO CONFERENCE ON SOFTWARE ENGINEERING AND ADVANCED APPLICATIONS (SEAA 2021): IEEE, s. 300-3079781665427050

Av författaren/redakt...: Liu, Yuchu, 1992; Issa Mattos, Dav ...; Bosch, Jan, 1967; Olsson, Helena H ...; Lantz, Jonn

Om ämnet

NATURVETENSKAP: NATURVETENSKAP; och Data och informa ...; och Programvarutekni ...

Kungliga biblioteket hanterar dina personuppgifter i enlighet med EU:s dataskyddsförordning (2018), GDPR. Läs mer om hur det funkar här.
Så här hanterar KB dina uppgifter vid användning av denna tjänst.