| 1. |
- Marculescu, Bogdan, et al.
(författare)
-
An initial industrial evaluation of interactive search-based testing for embedded software
- 2015
-
Ingår i: Applied Soft Computing. - : Elsevier. - 1568-4946 .- 1872-9681. ; 29, s. 26-39
-
Tidskriftsartikel (refereegranskat)abstract
- Search-based software testing promises the ability to generate and evaluate large numbers of test cases at minimal cost. From an industrial perspective, this could enable an increase in product quality without a matching increase in the time and effort required to do so. Search-based software testing, however, is a set of quite complex techniques and approaches that do not immediately translate into a process for use with most companies. For example, even if engineers receive the proper education and training in these new approaches, it can be hard to develop a general fitness function that covers all contingencies. Furthermore, in industrial practice, the knowledge and experience of domain specialists are often key for effective testing and thus for the overall quality of the final software system. But it is not clear how such domain expertise can be utilized in a search-based system. This paper presents an interactive search-based software testing (ISBST) system designed to operate in an industrial setting and with the explicit aim of requiring only limited expertise in software testing. It uses SBST to search for test cases for an industrial software module, while also allowing domain specialists to use their experience and intuition to interactively guide the search. In addition to presenting the system, this paper reports on an evaluation of the system in a company developing a framework for embedded software controllers. A sequence of workshops provided regular feedback and validation for the design and improvement of the ISBST system. Once developed, the ISBST system was evaluated by four electrical and system engineers from the company (the ’domain specialists’ in this context) used the system to develop test cases for a commonly used controller module. As well as evaluating the utility of the ISBST system, the study generated interaction data that were used in subsequent laboratory experimentation to validate the underlying search-based algorithm in the presence of realistic, but repeatable, interactions. The results validate the importance that automated software testing tools in general, and search-based tools, in particular, can leverage input from domain specialists while generating tests. Furthermore, the evaluation highlighted benefits of using such an approach to explore areas that the current testing practices do not cover or cover insufficiently. © 2014 Elsevier B.V. All rights reserved.
|
|
| 2. |
- Marculescu, Bogdan, et al.
(författare)
-
Tester interactivity makes a difference in search-based software testing : A controlled experiment
- 2016
-
Ingår i: Information and Software Technology. - : Elsevier. - 0950-5849 .- 1873-6025. ; 78, s. 66-82
-
Tidskriftsartikel (refereegranskat)abstract
- Context: Search-based software testing promises to provide users with the ability to generate high quality test cases, and hence increase product quality, with a minimal increase in the time and effort required. The development of the Interactive Search-Based Software Testing (ISBST) system was motivated by a previous study to investigate the application of search-based software testing (SBST) in an industrial setting. ISBST allows users to interact with the underlying SBST system, guiding the search and assessing the results. An industrial evaluation indicated that the ISBST system could find test cases that are not created by testers employing manual techniques. The validity of the evaluation was threatened, however, by the low number of participants. Objective: This paper presents a follow-up study, to provide a more rigorous evaluation of the ISBST system. Method: To assess the ISBST system a two-way crossover controlled experiment was conducted with 58 students taking a Verification and Validation course. The NASA Task Load Index (NASA-TLX) is used to assess the workload experienced by the participants in the experiment. Results:The experimental results validated the hypothesis that the ISBST system generates test cases that are not found by the same participants employing manual testing techniques. A follow-up laboratory experiment also investigates the importance of interaction in obtaining the results. In addition to this main result, the subjective workload was assessed for each participant by means of the NASA-TLX tool. The evaluation showed that, while the ISBST system required more effort from the participants, they achieved the same performance. Conclusions: The paper provides evidence that the ISBST system develops test cases that are not found by manual techniques, and that interaction plays an important role in achieving that result. (C) 2016 Elsevier B.V. All rights reserved.
|
|
| 3. |
- Marculescu, Bogdan, 1984, et al.
(författare)
-
Transferring interactive search-based software testing to industry
- 2018
-
Ingår i: Journal of Systems and Software. - : Elsevier BV. - 0164-1212 .- 1873-1228. ; 142, s. 156-170
-
Tidskriftsartikel (refereegranskat)abstract
- Context: Search-Based Software Testing (SBST), and the wider area of Search-Based Software Engineering (SBSE), is the application of optimization algorithms to problems in software testing, and software engineering, respectively. New algorithms, methods, and tools are being developed and validated on benchmark problems. In previous work, we have also implemented and evaluated Interactive Search-Based Software Testing (ISBST) tool prototypes, with a goal to successfully transfer the technique to industry. Objective: While SBST and SBSE solutions are often validated on benchmark problems, there is a need to validate them in an operational setting, and to assess their performance in practice. The present paper discusses the development and deployment of SBST tools for use in industry, and reflects on the transfer of these techniques to industry. Method: In addition to previous work discussing the development and validation of an ISBST prototype, a new version of the prototype ISBST system was evaluated in the laboratory and in industry. This evaluation is based on an industrial System under Test (SUT) and was carried out with industrial practitioners. The Technology Transfer Model is used as a framework to describe the progression of the development and evaluation of the ISBST system, as it progresses through the first five of its seven steps. Results: The paper presents a synthesis of previous work developing and evaluating the ISBST prototype, as well as presenting an evaluation, in both academia and industry, of that prototype's latest version. In addition to the evaluation, the paper also discusses the lessons learned from this transfer. Conclusions: This paper presents an overview of the development and deployment of the ISBST system in an industrial setting, using the framework of the Technology Transfer Model. We conclude that the ISBST system is capable of evolving useful test cases for that setting, though improvements in the means the system uses to communicate that information to the user are still required. In addition, a set of lessons learned from the project are listed and discussed. Our objective is to help other researchers that wish to validate search-based systems in industry, and provide more information about the benefits and drawbacks of these systems.
|
|
