| 1. |
|
|
| 2. |
|
|
| 3. |
- Clark, David, et al.
(författare)
-
Information Transformation : An Underpinning Theory for Software Engineering
- 2015
-
Ingår i: 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, Vol 2. - : IEEE. - 9781479919345 ; , s. 599-602
-
Konferensbidrag (refereegranskat)abstract
- Software engineering lacks underpinning scientific theories both for the software it produces and the processes by which it does so. We propose that an approach based on information theory can provide such a theory, or rather many theories. We envision that such a benefit will be realised primarily through research based on the quantification of information involved and a mathematical study of the limiting laws that arise. However, we also argue that less formal but more qualitative uses for information theory will be useful. The main argument in support of our vision is based on the fact that both a program and an engineering process to develop such a program are fundamentally processes that transform information. To illustrate our argument we focus on software testing and develop an initial theory in which a test suite is input/output adequate if it achieves the channel capacity of the program as measured by the mutual information between its inputs and its outputs. We outline a number of problems, metrics and concrete strategies for improving software engineering, based on information theoretical analyses. We find it likely that similar analyses and subsequent future research to detail them would be generally fruitful for software engineering.
|
|
| 4. |
- Feldt, Robert, 1972, et al.
(författare)
-
Searching for test data with feature diversity
- 2017
-
Tidskriftsartikel (övrigt vetenskapligt/konstnärligt)abstract
- There is an implicit assumption in software testing that more diverse and varied test data is needed for effective testing and to achieve different types and levels of coverage. Generic approaches based on information theory to measure and thus, implicitly, to create diverse data have also been proposed. However, if the tester is able to identify features of the test data that are important for the particular domain or context in which the testing is being performed, the use of generic diversity measures such as this may not be sufficient nor efficient for creating test inputs that show diversity in terms of these features. Here we investigate different approaches to find data that are diverse according to a specific set of features, such as length, depth of recursion etc. Even though these features will be less general than measures based on information theory, their use may provide a tester with more direct control over the type of diversity that is present in the test data. Our experiments are carried out in the context of a general test data generation framework that can generate both numerical and highly structured data. We compare random sampling for feature-diversity to different approaches based on search and find a hill climbing search to be efficient. The experiments highlight many trade-offs that needs to be taken into account when searching for diversity. We argue that recurrent test data generation motivates building statistical models that can then help to more quickly achieve feature diversity.
|
|
| 5. |
- Feldt, Robert, et al.
(författare)
-
Test Set Diameter : Quantifying the Diversity of Sets of Test Cases
- 2016
-
Ingår i: Proceedings - 2016 IEEE International Conference on Software Testing, Verification and Validation, ICST. - : IEEE Computer Society. - 9781509018260 ; , s. 223-233
-
Konferensbidrag (refereegranskat)abstract
- A common and natural intuition among software testers is that test cases need to differ if a software system is to be tested properly and its quality ensured. Consequently, much research has gone into formulating distance measures for how test cases, their inputs and/or their outputs differ. However, common to these proposals is that they are data type specific and/or calculate the diversity only between pairs of test inputs, traces or outputs. We propose a new metric to measure the diversity of sets of tests: the test set diameter (TSDm). It extends our earlier, pairwise test diversity metrics based on recent advances in information theory regarding the calculation of the normalized compression distance (NCD) for multisets. A key advantage is that TSDm is a universal measure of diversity and so can be applied to any test set regardless of data type of the test inputs (and, moreover, to other test-related data such as execution traces). But this universality comes at the cost of greater computational effort compared to competing approaches. Our experiments on four different systems show that the test set diameter can help select test sets with higher structural and fault coverage than random selection even when only applied to test inputs. This can enable early test design and selection, prior to even having a software system to test, and complement other types of test automation and analysis. We argue that this quantification of test set diversity creates a number of opportunities to better understand software quality and provides practical ways to increase it.
|
|
| 6. |
- Irshad, Mohsin, et al.
(författare)
-
A systematic literature review of software requirements reuse approaches
- 2018
-
Ingår i: Information and Software Technology. - : Elsevier. - 0950-5849 .- 1873-6025. ; 93:Jan, s. 223-245
-
Forskningsöversikt (refereegranskat)abstract
- Context: Early software reuse is considered as the most beneficial form of software reuse. Hence, previous research has focused on supporting the reuse of software requirements. Objective: This study aims to identify and investigate the current state of the art with respect to (a) what requirement reuse approaches have been proposed, (b) the methods used to evaluate the approaches, (c) the characteristics of the approaches, and (d) the quality of empirical studies on requirements reuse with respect to rigor and relevance. Method: We conducted a systematic review and a combination of snowball sampling and database search have been used to identify the studies. The rigor and relevance scoring rubric has been used to assess the quality of the empirical studies. Multiple researchers have been involved in each step to increase the reliability of the study. Results: Sixty-nine studies were identified that describe requirements reuse approaches. The majority of the approaches used structuring and matching of requirements as a method to support requirements reuse and text-based artefacts were commonly used as an input to these approaches. Further evaluation of the studies revealed that the majority of the approaches are not validated in the industry. The subset of empirical studies (22 in total) was analyzed for rigor and relevance and two studies achieved the maximum score for rigor and relevance based on the rubric. It was found that mostly text-based requirements reuse approaches were validated in the industry. Conclusion: From the review, it was found that a number of approaches already exist in literature, but many approaches are not validated in industry. The evaluation of rigor and relevance of empirical studies show that these do not contain details of context, validity threats, and the industrial settings, thus highlighting the need for the industrial evaluation of the approaches. © 2017 Elsevier B.V.
|
|
| 7. |
- Marculescu, Bogdan, et al.
(författare)
-
An initial industrial evaluation of interactive search-based testing for embedded software
- 2015
-
Ingår i: Applied Soft Computing. - : Elsevier. - 1568-4946 .- 1872-9681. ; 29, s. 26-39
-
Tidskriftsartikel (refereegranskat)abstract
- Search-based software testing promises the ability to generate and evaluate large numbers of test cases at minimal cost. From an industrial perspective, this could enable an increase in product quality without a matching increase in the time and effort required to do so. Search-based software testing, however, is a set of quite complex techniques and approaches that do not immediately translate into a process for use with most companies. For example, even if engineers receive the proper education and training in these new approaches, it can be hard to develop a general fitness function that covers all contingencies. Furthermore, in industrial practice, the knowledge and experience of domain specialists are often key for effective testing and thus for the overall quality of the final software system. But it is not clear how such domain expertise can be utilized in a search-based system. This paper presents an interactive search-based software testing (ISBST) system designed to operate in an industrial setting and with the explicit aim of requiring only limited expertise in software testing. It uses SBST to search for test cases for an industrial software module, while also allowing domain specialists to use their experience and intuition to interactively guide the search. In addition to presenting the system, this paper reports on an evaluation of the system in a company developing a framework for embedded software controllers. A sequence of workshops provided regular feedback and validation for the design and improvement of the ISBST system. Once developed, the ISBST system was evaluated by four electrical and system engineers from the company (the ’domain specialists’ in this context) used the system to develop test cases for a commonly used controller module. As well as evaluating the utility of the ISBST system, the study generated interaction data that were used in subsequent laboratory experimentation to validate the underlying search-based algorithm in the presence of realistic, but repeatable, interactions. The results validate the importance that automated software testing tools in general, and search-based tools, in particular, can leverage input from domain specialists while generating tests. Furthermore, the evaluation highlighted benefits of using such an approach to explore areas that the current testing practices do not cover or cover insufficiently. © 2014 Elsevier B.V. All rights reserved.
|
|
| 8. |
|
|
| 9. |
- Marculescu, Bogdan, et al.
(författare)
-
Tester interactivity makes a difference in search-based software testing : A controlled experiment
- 2016
-
Ingår i: Information and Software Technology. - : Elsevier. - 0950-5849 .- 1873-6025. ; 78, s. 66-82
-
Tidskriftsartikel (refereegranskat)abstract
- Context: Search-based software testing promises to provide users with the ability to generate high quality test cases, and hence increase product quality, with a minimal increase in the time and effort required. The development of the Interactive Search-Based Software Testing (ISBST) system was motivated by a previous study to investigate the application of search-based software testing (SBST) in an industrial setting. ISBST allows users to interact with the underlying SBST system, guiding the search and assessing the results. An industrial evaluation indicated that the ISBST system could find test cases that are not created by testers employing manual techniques. The validity of the evaluation was threatened, however, by the low number of participants. Objective: This paper presents a follow-up study, to provide a more rigorous evaluation of the ISBST system. Method: To assess the ISBST system a two-way crossover controlled experiment was conducted with 58 students taking a Verification and Validation course. The NASA Task Load Index (NASA-TLX) is used to assess the workload experienced by the participants in the experiment. Results:The experimental results validated the hypothesis that the ISBST system generates test cases that are not found by the same participants employing manual testing techniques. A follow-up laboratory experiment also investigates the importance of interaction in obtaining the results. In addition to this main result, the subjective workload was assessed for each participant by means of the NASA-TLX tool. The evaluation showed that, while the ISBST system required more effort from the participants, they achieved the same performance. Conclusions: The paper provides evidence that the ISBST system develops test cases that are not found by manual techniques, and that interaction plays an important role in achieving that result. (C) 2016 Elsevier B.V. All rights reserved.
|
|
| 10. |
- Marculescu, Bogdan, 1984, et al.
(författare)
-
Transferring interactive search-based software testing to industry
- 2018
-
Ingår i: Journal of Systems and Software. - : Elsevier BV. - 0164-1212 .- 1873-1228. ; 142, s. 156-170
-
Tidskriftsartikel (refereegranskat)abstract
- Context: Search-Based Software Testing (SBST), and the wider area of Search-Based Software Engineering (SBSE), is the application of optimization algorithms to problems in software testing, and software engineering, respectively. New algorithms, methods, and tools are being developed and validated on benchmark problems. In previous work, we have also implemented and evaluated Interactive Search-Based Software Testing (ISBST) tool prototypes, with a goal to successfully transfer the technique to industry. Objective: While SBST and SBSE solutions are often validated on benchmark problems, there is a need to validate them in an operational setting, and to assess their performance in practice. The present paper discusses the development and deployment of SBST tools for use in industry, and reflects on the transfer of these techniques to industry. Method: In addition to previous work discussing the development and validation of an ISBST prototype, a new version of the prototype ISBST system was evaluated in the laboratory and in industry. This evaluation is based on an industrial System under Test (SUT) and was carried out with industrial practitioners. The Technology Transfer Model is used as a framework to describe the progression of the development and evaluation of the ISBST system, as it progresses through the first five of its seven steps. Results: The paper presents a synthesis of previous work developing and evaluating the ISBST prototype, as well as presenting an evaluation, in both academia and industry, of that prototype's latest version. In addition to the evaluation, the paper also discusses the lessons learned from this transfer. Conclusions: This paper presents an overview of the development and deployment of the ISBST system in an industrial setting, using the framework of the Technology Transfer Model. We conclude that the ISBST system is capable of evolving useful test cases for that setting, though improvements in the means the system uses to communicate that information to the user are still required. In addition, a set of lessons learned from the project are listed and discussed. Our objective is to help other researchers that wish to validate search-based systems in industry, and provide more information about the benefits and drawbacks of these systems.
|
|
| 11. |
- Poulding, Simon, et al.
(författare)
-
Automated Random Testing in Multiple Dispatch Languages
- 2017
-
Ingår i: Proceedings - 10th IEEE International Conference on Software Testing, Verification and Validation, ICST 2017. - : IEEE Computer Society. - 9781509060313 ; , s. 333-344
-
Konferensbidrag (refereegranskat)abstract
- In programming languages that use multiple dispatch, a single function can have multiple implementations, each of which may specialise the function's operation. Which one of these implementations to execute is determined by the data types of all the arguments to the function. Effective testing of functions that use multiple dispatch therefore requires diverse test inputs in terms of the data types of the input's arguments as well as their values. In this paper we describe an approach for generating test inputs where both the values and types are chosen probabilistically. The approach uses reflection to automatically determine how to create inputs with the desired types, and dynamically updates the probability distribution from which types are sampled in order to improve both the test efficiency and efficacy. We evaluate the technique on 247 methods across 9 built-in functions of Julia, a technical computing language that applies multiple dispatch at runtime. In the process, we identify three real faults in these widely-used functions. © 2017 IEEE.
|
|
| 12. |
- Poulding, Simon, et al.
(författare)
-
Generating Controllably Invalid and Atypical Inputs for Robustness Testing
- 2017
-
Ingår i: Proceedings - 10th IEEE International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2017. - : Institute of Electrical and Electronics Engineers Inc.. - 9781509066766 ; , s. 81-84
-
Konferensbidrag (refereegranskat)abstract
- One form of robustness in a software system is its ability to handle, in an appropriate manner, inputs that are unexpected compared to those it would experience in normal operation. In this paper we investigate a generic approach to generating such unexpected test inputs by extending a framework that we have previously developed for the automated creation of complex and high-structured test data. The approach is applied to the generation of valid inputs that are atypical as well as inputs that are invalid. We demonstrate that our approach enables control of the 'degree' to which the test data is invalid or atypical, and show empirically that this can alter the extent to which the robustness of a software system is exercised during testing. © 2017 IEEE.
|
|
| 13. |
- Poulding, Simon, et al.
(författare)
-
Generating Structured Test Data with Specific Properties using Nested Monte-Carlo Search
- 2014
-
Ingår i: GECCO'14: PROCEEDINGS OF THE 2014 GENETIC AND EVOLUTIONARY COMPUTATION CONFERENCE. - New York, NY, USA : Association for Computing Machinery (ACM). - 9781450326629 ; , s. 1279-1286
-
Konferensbidrag (refereegranskat)abstract
- Software acting on complex data structures can be challenging to test: it is difficult to generate diverse test data that satisfies structural constraints while simultaneously exhibiting properties, such as a particular size, that the test engineer believes will be effective in detecting faults. In our previous work we introduced GödelTest, a framework for generating such data structures using non-deterministic programs, and combined it with Differential Evolution to optimize the generation process. Monte-Carlo Tree Search (MCTS) is a search technique that has shown great success in playing games that can be represented as sequence of decisions. In this paper we apply Nested Monte-Carlo Search, a single-player variant of MCTS, to the sequence of decisions made by the generating programs used by GödelTest, and show that this combination can efficiently generate random data structures which exhibit the specific properties that the test engineer requires. We compare the results to Boltzmann sampling, an analytical approach to generating random combinatorial data structures.
|
|
| 14. |
|
|
| 15. |
- Poulding, Simon, et al.
(författare)
-
Re-using generators of complex test data
- 2015
-
Ingår i: 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST). - : IEEE Computer Society. - 9781479971251 ; , s. Article number 7102605-
-
Konferensbidrag (refereegranskat)abstract
- The efficiency of random testing can be improved by sampling test inputs using a generating program that incorporates knowledge about the types of input most likely to detect faults in the software-under-test (SUT). But when the input of the SUT is a complex data type - such as a domain-specific string, array, record, tree, or graph - creating such a generator may be time- consuming and may require the tester to have substantial prior experience of the domain. In this paper we propose the re-use of generators created for one SUT on other SUTs that take the same complex data type as input. The re-use of a generator in this way would have little overhead, and we hypothesise that the re-used generator will typically be as least as efficient as the most straightforward form of random testing: sampling test inputs from the uniform distribution. We investigate this proposal for two data types using five generators. We assess test efficiency against seven real-world SUTs, and in terms of both structural coverage and the detection of seeded faults. The results support the re-use of generators for complex data types, and suggest that if a library of generators is to be maintained for this purpose, it is possible to extend library generators to accommodate the specific testing requirements of newly-encountered SUTs. © 2015 IEEE.
|
|
| 16. |
- Poulding, Simon, et al.
(författare)
-
The optimisation of stochastic grammars to enable cost-effective probabilistic structural testing
- 2015
-
Ingår i: Journal of Systems and Software. - : Elsevier. - 0164-1212 .- 1873-1228. ; 103, s. 296-310
-
Tidskriftsartikel (refereegranskat)abstract
- The effectiveness of statistical testing, a probabilistic structural testing strategy, depends on the characteristics of the probability distribution from which test inputs are sampled. Metaheuristic search has been shown to be a practical method of optimising the characteristics of such distributions. However, the applicability of the existing search-based algorithm is limited by the requirement that the software's inputs must be a fixed number of ordinal values. In this paper we propose a new algorithm that relaxes this limitation and so permits the derivation of probability distributions for a much wider range of software. The representation used by the new algorithm is based on a stochastic grammar supplemented with two novel features: conditional production weights and the dynamic partitioning of ordinal ranges. We demonstrate empirically that a search algorithm using this representation can optimise probability distributions over complex input domains and thereby enable costeffective statistical testing, and that the use of both conditional production weights and dynamic partitioning can be beneficial to the search process. (C) 2014 Elsevier Inc. All rights reserved.
|
|
| 17. |
- Poulding, Simon, et al.
(författare)
-
Using Citation Behavior to Rethink Academic Impact in Software Engineering
- 2015
-
Ingår i: ACM-IEEE International Symposium on Empirical Software Engineering and Measurement. - 9781467378994 ; , s. 140-43
-
Konferensbidrag (refereegranskat)abstract
- Although citation counts are often considered a measure of academic impact, they are criticized for failing to evaluate impact as intended. In this paper we propose that software engineering citations may be classified according to how the citation is used by the author of the citing paper, and that through this classification of citation behaviour it is possible to achieve a more refined understanding of the cited paper’s impact. Our objective in this work is to conduct an initial evaluation using the citation behaviour taxonomy proposed by Bornmann and Daniel. We independently classified citations to ten highly-cited papers published at the International Symposium on Empirical Software Engineering and Measurement (ESEM). The degree to which classifications were consistent between researchers was analyzed in order to assess the clarity of Bornmann and Daniel’s taxonomy. We found poor to fair agreement between researchers even though the taxonomy was perceived as relatively easy to apply for the majority of citations. We were nevertheless able to identify clear differences in the profile of citation behaviors between the cited papers. We conclude that an improved taxonomy is required if classification is to be reliable, and that a degree of automation would improve reliability as well as reduce the time taken to make a classification.
|
|
| 18. |
- Rose, Louise, et al.
(författare)
-
Towards a scalable cloud platform for search-based probabilistic testing
- 2013
-
Konferensbidrag (refereegranskat)abstract
- Probabilistic testing techniques that sample input data at random from a probability distribution can be more effective at detecting faults than deterministic techniques. However, if overly large (and therefore expensive) test sets are to be avoided, the probability distribution from which the input data is sampled must be optimised to the particular software-under-test. Such an optimisation process is often resource-intensive. In this paper, we present a prototypical cloud platform-and architecture-that permits the optimisation of such probability distributions in a scalable, distributed and robust manner, and thereby enables cost-effective probabilistic testing.
|
|
| 19. |
|
|
