| 1. |
- Felderer, Michael, 1978-, et al.
(author)
-
A testability analysis framework for non-functional properties
- 2018
-
In: 2018 IEEE 11TH INTERNATIONAL CONFERENCE ON SOFTWARE TESTING, VERIFICATION AND VALIDATION WORKSHOPS (ICSTW). - : Institute of Electrical and Electronics Engineers Inc.. - 9781538663523 ; , s. 54-58
-
Conference paper (peer-reviewed)abstract
- This paper presents background, the basic steps and an example for a testability analysis framework for non-functional properties.
|
|
| 2. |
- Feldt, Robert, et al.
(author)
-
Optimizing Verification and Validation Activities for Software in the Space Industry
- 2010
-
In: European Space Agency, (Special Publication) ESA SP. Conference on DAta Systems in Aerospace, DASIA 2010; Budapest; 1-4 June 2010. - Budapest : European Space Agency. - 0379-6566. - 9789290922469 ; , s. 309-313
-
Conference paper (peer-reviewed)abstract
- Software for space applications has special requirements in terms of reliability and dependability and the verification & validation activities (VAs) of these systems often account for more than 50% of the develop- ment effort. The industry is also faced with political and market pressure to deliver software faster and cheaper. Thus new ways are needed to optimize these activities so that high quality can be retained even with reduced costs and effort. Here we present a framework for the management and optimization of verification & validation activities (VAMOS). An initial evaluation of the framework based on historical data as well as data extracted with a new tool has been done and are described briefly.
|
|
| 3. |
- Marculescu, Bogdan, et al.
(author)
-
A concept for an interactive search-based software testing system
- 2012
-
In: Lecture Notes in Computer Science. - Berlin, Heidelberg : Springer. ; , s. 273-278
-
Conference paper (peer-reviewed)abstract
- Software is an increasingly important part of various products, although not always the dominant component. For these software-intensive systems it is common that the software is assembled, and sometimes even developed, by domain specialists rather than by software engineers. To leverage the domain specialists' knowledge while maintaining quality we need testing tools that require only limited knowledge of software testing. Since each domain has unique quality criteria and trade-offs and there is a large variation in both software modeling and implementation syntax as well as semantics it is not easy to envisage general software engineering support for testing tasks. Particularly not since such support must allow interaction between the domain specialists and the testing system for iterative development. In this paper we argue that search-based software testing can provide this type of general and interactive testing support and describe a proof of concept system to support this argument. The system separates the software engineering concerns from the domain concerns and allows domain specialists to interact with the system in order to select the quality criteria being used to determine the fitness of potential solutions.
|
|
| 4. |
- Marculescu, Bogdan, et al.
(author)
-
An initial industrial evaluation of interactive search-based testing for embedded software
- 2015
-
In: Applied Soft Computing. - : Elsevier. - 1568-4946 .- 1872-9681. ; 29, s. 26-39
-
Journal article (peer-reviewed)abstract
- Search-based software testing promises the ability to generate and evaluate large numbers of test cases at minimal cost. From an industrial perspective, this could enable an increase in product quality without a matching increase in the time and effort required to do so. Search-based software testing, however, is a set of quite complex techniques and approaches that do not immediately translate into a process for use with most companies. For example, even if engineers receive the proper education and training in these new approaches, it can be hard to develop a general fitness function that covers all contingencies. Furthermore, in industrial practice, the knowledge and experience of domain specialists are often key for effective testing and thus for the overall quality of the final software system. But it is not clear how such domain expertise can be utilized in a search-based system. This paper presents an interactive search-based software testing (ISBST) system designed to operate in an industrial setting and with the explicit aim of requiring only limited expertise in software testing. It uses SBST to search for test cases for an industrial software module, while also allowing domain specialists to use their experience and intuition to interactively guide the search. In addition to presenting the system, this paper reports on an evaluation of the system in a company developing a framework for embedded software controllers. A sequence of workshops provided regular feedback and validation for the design and improvement of the ISBST system. Once developed, the ISBST system was evaluated by four electrical and system engineers from the company (the ’domain specialists’ in this context) used the system to develop test cases for a commonly used controller module. As well as evaluating the utility of the ISBST system, the study generated interaction data that were used in subsequent laboratory experimentation to validate the underlying search-based algorithm in the presence of realistic, but repeatable, interactions. The results validate the importance that automated software testing tools in general, and search-based tools, in particular, can leverage input from domain specialists while generating tests. Furthermore, the evaluation highlighted benefits of using such an approach to explore areas that the current testing practices do not cover or cover insufficiently. © 2014 Elsevier B.V. All rights reserved.
|
|
| 5. |
- Marculescu, Bogdan, et al.
(author)
-
Finding a Boundary between Valid and Invalid Regions of the Input Space
- 2018
-
In: Proceedings - Asia-Pacific Software Engineering Conference, APSEC. - : IEEE Computer Society. - 9781728119700 ; 2018-December, s. 169-178
-
Conference paper (peer-reviewed)abstract
- In the context of robustness testing, the boundary between the valid and invalid regions of the input space can be an interesting source of erroneous inputs. Knowing where a specific software under test (SUT) has a boundary is also essential for validation in relation to requirements. However, finding where a SUT actually implements the boundary is a non-trivial problem that has not gotten much attention. This paper proposes a method of finding the boundary between the valid and invalid regions of the input space, by developing pairs of test sets that describe that boundary in detail. The proposed method consists of two steps. First, test data generators, directed by a search algorithm to maximise distance to known, valid test cases, generate valid test cases that are closer to the boundary. Second, these valid test cases undergo mutations to try to push them over the boundary and into the invalid part of the input space. This results in a pair of test sets, one consisting of test cases on the valid side of the boundary and a matched set on the outer side, with only a small distance between the two sets. The method is evaluated on a number of examples from the standard library of a modern programming language. We propose a method of determining the boundary between valid and invalid regions of the input space, and apply it on a SUT that has a non-contiguous valid region of the input space. From the small distance between the developed pairs of test sets, and the fact that one test set contains valid test cases and the other invalid test cases, we conclude that the pair of test sets described the boundary between the valid and invalid regions of that input space. Differences of behaviour can be observed between different distances and different sets of mutation operators, but all show that the method is able to identify the boundary between the valid and invalid regions of the input space. This is an important step towards more automated robustness testing. © 2018 IEEE.
|
|
| 6. |
- Marculescu, Bogdan, et al.
(author)
-
Flexible and Low-Cost Measurements for Space Software Development : The Measurements Exploration Framework
- 2011
-
Conference paper (peer-reviewed)abstract
- Verification and validation is an important part of software development and accounts for significant amounts of the costs associated with such a project. For developers of life or mission critical systems, such as software being developed for space applications, a balance must be reached between ensuring the quality of the system by extensive and rigorous testing and reducing costs and allowing the com- pany to compete. Ensuring the quality of any system starts with a quality development process. To evaluate both the software development process and the product itself, measurements are needed. A balance must be then struck between ensuring the best possible quality of both process and product on the one hand, and reducing the cost of performing requirements on the other. A number of measurements have already been defined and are being used. For some of these, data collection can be automated as well, further lowering costs associated with implementing them. In practice, however, there may be situations where existing measurements are unsuitable for a variety of reasons. This paper describes a framework for creating low cost, flexible measurements in areas where initial information is scarce. The framework, called The Measurements Exploration Framework, is aimed in particular at the Space Software development industry and was developed is such an environment.
|
|
| 7. |
|
|
| 8. |
- Marculescu, Bogdan
(author)
-
Interactive Search-Based Testing of Embedded Software : Understanding the Benefits and Difficulties of Industrial Application
- 2014
-
Licentiate thesis (other academic/artistic)abstract
- The ubiquity of software has wide-ranging consequences for its development and testing. Increasingly often, software is developed and tested by engineers specialized in other areas. Embedded software, for example, is developed ad-hoc, for each product, by systems and domain engineers. Supporting testing activities in this context requires a highly flexible approach, powerful enough to create useful test cases, yet simple enough to not require specialized training in software testing. Search-based software testing promises the ability to generate and evaluate large numbers of test cases at minimal cost. It is, however, a set of complex techniques that cannot be used off-the-shelf as part of the software development process of a company. The objective of the work presented in this thesis is to investigate the applicability of Search-Based Software Testing in an industrial environment. A second objective was identifying additional knowledge gaps relating to integrating such techniques in existing software development processes. Investigating how meaningful interaction is to take place, what information is needed in order for both stakeholders to be able to achieve their objectives is a third goal. The findings are obtained by means of a series of case studies in a company developing both embedded software and the tools to enable embedded software development. A prototype Interactive Search-Based Software Testing (ISBST) system was developed that uses user interaction to drive the search-based component towards interesting test cases. The ISBST system was evaluated constantly, and improved based on the findings of each case study. The latest case study was an empirical evaluation of the system with the engineers, both software engineers and domain specialists, in the company. The empirical work includes both qualitative and quantitative data, with a focus on the exploratory study of the practical factors affecting the use of the ISBST system. A key early finding is that interactivity is essential when implementing search-based techniques in the industrial context described above. Repeated validations conducted with the company yielded additional information on the practicalities of interaction. The strength of SBST proved useful in investigating areas of the test space that were normally overlooked due to limitations in terms of resources. At the same time, developers were able to use their experience and intuition to guide the SBST system towards test cases that were more likely to be problematic. Overall, the results obtained indicate that the search-based techniques provide a useful complement to existing testing techniques. SBST, in its interactive form, can be a useful complement to existing testing techniques. An Interactive SBST (ISBST) system has been developed as a result of this research. Results show that this system is usable by the developers of embedded software, that often specialize on acquiring domain knowledge rather than software engineering expertise.
|
|
| 9. |
- Marculescu, Bogdan, 1984, et al.
(author)
-
Objective re-weighting to guide an interactive search based software testing system
- 2013
-
In: 2013 12th International Conference on Machine Learning and Applications, ICMLA 2013; Miami, FL; United States; 4 December 2013 through 7 December 2013. - Miami : IEEE. ; 2, s. 102-107
-
Conference paper (peer-reviewed)abstract
- Even hardware-focused industries today develop products where software is both a large and important component. Engineers tasked with developing and integrating these products do not always have a software engineering background. To ensure quality, tools are needed that automate and support software testing while allowing these domain specialists to leverage their knowledge and experience. Search-based testing could be a key aspect in creating an automated tool for supporting testing activities. However, domain specific quality criteria and trade-offs make it difficult to develop a general fitness function a priori, so interaction between domain specialists and such a tool would be critical to its success. In this paper we present a system for interactive search based software testing and investigate a way for domain specialists to guide the search by dynamically re-weighting quality goals. Our empirical investigation shows that objective reweighing can help a human domain specialist interactively guide the search, without requiring specialised knowledge of the system and without sacrificing population diversity.
|
|
| 10. |
- Marculescu, Bogdan, et al.
(author)
-
Perception of Scientific Evidence: Do Industry and Academia Share an Understanding?
- 2016
-
Conference paper (peer-reviewed)abstract
- Context: Collaboration depends on communication and upon having a similar understanding of the notions that are being discussed, and a similar appraisal of their value. Existing work seems to show that the collaboration between industry and academia is hampered by a difference in values. In particular, academic work focuses more on generalizing on the basis of existing evidence, while industry prefers to particularize conclusions to individual cases. This has lead to the conclusion that industry values scientific evidence less than academia does. Objective: This paper seeks to re-evaluate that conclusion, and investigate if industry and academia share a definition of scientific evidence. If evidence can be found of competing views, we propose a more finely grained model of empirical evidence and its role in building software engineering knowledge. Moreover, we seek to determine if a more nuanced look the notion of scientific evidence has an influence on how academics and industry practitioners perceive that notion. Method: We have developed a model of key concepts related to understanding empirical evidence in software engineering. An initial validation has been conducted, consisting of a survey of master students, to determine if competing views of evidence exist at that level. The model will be validated by further literature study and semistructured interviews with industry practitioners. Results: We propose a model of empirical evidence in software engineering, and initial validation of that model by means of a survey. The results of the survey indicate that conflicting opinions already exist in the student body regarding the notion of evidence, how trustworthy different sources of evidence and knowledge are, and which sources of evidence and types of evidence are more appropriate in various situations.Conclusion: Rather than a difference in how industry and academia value scientific evidence, we see evidence of misunderstanding, of different notions of what constitutes scientific evidence and what strength of evidence is required to achieve specific goals. We propose a model of empirical evidence, to provide a better understanding of what is required in various situations and a better platform for communication between industry and academia.
|
|
| 11. |
- Marculescu, Bogdan, et al.
(author)
-
Practitioner-Oriented Visualization in an Interactive Search-Based Software Test Creation Tool
- 2013
-
In: Proceedings - Asia-Pacific Software Engineering Conference, APSEC. - : IEEE Computer Society. - 1530-1362. - 9781479921430 ; 2, s. 87-92
-
Conference paper (peer-reviewed)abstract
- Search-based software testing uses meta-heuristic search techniques to automate or partially automate testing tasks, such as test case generation or test data generation. It uses a fitness function to encode the quality characteristics that are relevant, for a given problem, and guides the search to acceptable solutions in a potentially vast search space. From an industrial perspective, this opens up the possibility of generating and evaluating lots of test cases without raising costs to unacceptable levels. First, however, the applicability of search-based software engineering in an industrial setting must be evaluated. In practice, it is difficult to develop a priori a fitness function that covers all practical aspects of a problem. Interaction with human experts offers access to experience that is otherwise unavailable and allows the creation of a more informed and accurate fitness function. Moreover, our industrial partner has already expressed a view that the knowledge and experience of domain specialists are more important to the overall quality of the systems they develop than software engineering expertise. In this paper we describe our application of Interactive Search Based Software Testing (ISBST) in an industrial setting. We used SBST to search for test cases for an industrial software module and based, in part, on interaction with a human domain specialist. Our evaluation showed that such an approach is feasible, though it also identified potential difficulties relating to the interaction between the domain specialist and the system.
|
|
| 12. |
- Marculescu, Bogdan, et al.
(author)
-
Tester interactivity makes a difference in search-based software testing : A controlled experiment
- 2016
-
In: Information and Software Technology. - : Elsevier. - 0950-5849 .- 1873-6025. ; 78, s. 66-82
-
Journal article (peer-reviewed)abstract
- Context: Search-based software testing promises to provide users with the ability to generate high quality test cases, and hence increase product quality, with a minimal increase in the time and effort required. The development of the Interactive Search-Based Software Testing (ISBST) system was motivated by a previous study to investigate the application of search-based software testing (SBST) in an industrial setting. ISBST allows users to interact with the underlying SBST system, guiding the search and assessing the results. An industrial evaluation indicated that the ISBST system could find test cases that are not created by testers employing manual techniques. The validity of the evaluation was threatened, however, by the low number of participants. Objective: This paper presents a follow-up study, to provide a more rigorous evaluation of the ISBST system. Method: To assess the ISBST system a two-way crossover controlled experiment was conducted with 58 students taking a Verification and Validation course. The NASA Task Load Index (NASA-TLX) is used to assess the workload experienced by the participants in the experiment. Results:The experimental results validated the hypothesis that the ISBST system generates test cases that are not found by the same participants employing manual testing techniques. A follow-up laboratory experiment also investigates the importance of interaction in obtaining the results. In addition to this main result, the subjective workload was assessed for each participant by means of the NASA-TLX tool. The evaluation showed that, while the ISBST system required more effort from the participants, they achieved the same performance. Conclusions: The paper provides evidence that the ISBST system develops test cases that are not found by manual techniques, and that interaction plays an important role in achieving that result. (C) 2016 Elsevier B.V. All rights reserved.
|
|
| 13. |
- Marculescu, Bogdan, 1984, et al.
(author)
-
Transferring interactive search-based software testing to industry
- 2018
-
In: Journal of Systems and Software. - : Elsevier BV. - 0164-1212 .- 1873-1228. ; 142, s. 156-170
-
Journal article (peer-reviewed)abstract
- Context: Search-Based Software Testing (SBST), and the wider area of Search-Based Software Engineering (SBSE), is the application of optimization algorithms to problems in software testing, and software engineering, respectively. New algorithms, methods, and tools are being developed and validated on benchmark problems. In previous work, we have also implemented and evaluated Interactive Search-Based Software Testing (ISBST) tool prototypes, with a goal to successfully transfer the technique to industry. Objective: While SBST and SBSE solutions are often validated on benchmark problems, there is a need to validate them in an operational setting, and to assess their performance in practice. The present paper discusses the development and deployment of SBST tools for use in industry, and reflects on the transfer of these techniques to industry. Method: In addition to previous work discussing the development and validation of an ISBST prototype, a new version of the prototype ISBST system was evaluated in the laboratory and in industry. This evaluation is based on an industrial System under Test (SUT) and was carried out with industrial practitioners. The Technology Transfer Model is used as a framework to describe the progression of the development and evaluation of the ISBST system, as it progresses through the first five of its seven steps. Results: The paper presents a synthesis of previous work developing and evaluating the ISBST prototype, as well as presenting an evaluation, in both academia and industry, of that prototype's latest version. In addition to the evaluation, the paper also discusses the lessons learned from this transfer. Conclusions: This paper presents an overview of the development and deployment of the ISBST system in an industrial setting, using the framework of the Technology Transfer Model. We conclude that the ISBST system is capable of evolving useful test cases for that setting, though improvements in the means the system uses to communicate that information to the user are still required. In addition, a set of lessons learned from the project are listed and discussed. Our objective is to help other researchers that wish to validate search-based systems in industry, and provide more information about the benefits and drawbacks of these systems.
|
|
| 14. |
- Marculescu, Bogdan, 1984, et al.
(author)
-
Using exploration focused techniques to augment search-based software testing: an experimental evaluation
- 2016
-
In: 2016 IEEE International Conference on Software Testing, Verification and Validation (ICST). - : IEEE. - 2381-2834. - 9781509018277 - 9780306406157 ; , s. 69-79
-
Conference paper (peer-reviewed)abstract
- Search-based software testing (SBST)often uses objective-based approaches to solve testing problems. There are, however, situations where the validity and completeness of objectives cannot be ascertained, or where there is insufficient information to define objectives at all. Incomplete or incorrect objectives may steer the search away from interesting behavior of the software under test (SUT) and from potentially useful test cases.This papers investigates the degree to whichexploration-based algorithms can be used tocomplement an objective-based tool we have previously developed and evaluated in industry. In particular, we would like to assess how exploration-based algorithms perform in situations where little information on the behavior space is available a priori. We have conducted an experiment comparing the performance of an exploration-based algorithm with an objective-based one on a problem with a high dimensional behavior space. In addition, we evaluate to what extent that performance degrades in situations where computational resources are limited.Our experiment shows that exploration-basedalgorithms are useful in covering a larger area of the behavior space and result in a more diverse solution population. Typically, of the candidate solutions that exploration-based algorithms propose, more than 80% were not covered by their objective-based counterpart. This increased diversity is present in the resulting population even when computational resources are limited. We conclude that exploration-focused algorithms are a useful means of investigating high-dimensional spaces, even in situations where limited information and limited resources are available.
|
|
