| 1. |
- Grindal, Mats, et al.
(författare)
-
An Evaluation of Combination Strategies for Test Case Selection
- 2003
-
Rapport (övrigt vetenskapligt)abstract
- In this report we present the results from a comparative evaluation of five combination strategies. Combination strategies are test case selection methods that combine interesting values of the input parameters of a test object to form test cases. One of the investigated combination strategies, namely the Each Choice strategy, satisfies 1-wise coverage, i.e., each interesting value of each parameter is represented at least once in the test suite. Two of the strategies, the Orthogonal Arrays and Heuristic Pair-Wise strategies both satisfy pair-wise coverage, i.e., every possible pair of interesting values of any two parameters are included in the test suite. The fourth combination strategy, the All Values strategy, generates all possible combinations of the interesting values of the input parameters. The fifth and last combination strategy, the Base Choice combination strategy, satisfies 1-wise coverage but in addition makes use of some semantic information to construct the test cases.Except for the All Values strategy, which is only used as a reference point with respect to the number of test cases, the combination strategies are evaluated and compared with respect to number of test cases, number of faults found, test suite failure density, and achieved decision coverage in an experiment comprising five programs, similar to Unix commands, seeded with 131 faults. As expected, the Each Choice strategy finds the smallest number of faults among the evaluated combination strategies. Surprisingly, the Base Choice strategy performs as well, in terms of detecting faults, as the pair-wise combination strategies, despite fewer test cases. Since the programs and faults in our experiment may not be representative of actual testing problems in an industrial setting, we cannot draw any general conclusions regarding the number of faults detected by the evaluated combination strategies. However, our analysis shows some properties of the combination strategies that appear significant in spite of the programs and faults not being representative. The two most important results are that the Each Choice strategy is unpredictable in terms of which faults will be detected, i.e., most faults found are found by chance, and that the Base Choice and the pair-wise combination strategies to some extent target different types of faults.
|
|
| 2. |
- Grindal, Mats, et al.
(författare)
-
An Evaluation of Combination Strategies for Test Case Selection
- 2006
-
Ingår i: Journal of Empirical Software Engineering. - Springer. - 1382-3256. ; 11:4, s. 583-611
-
Tidskriftsartikel (refereegranskat)abstract
- This paper presents results from a comparative evaluation of five combination strategies. Combination strategies are test case selection methods that combine “interesting” values of the input parameters of a test subject to form test cases. This research comparatively evaluated five combination strategies; the All Combination strategy (AC), the Each Choice strategy (EC), the Base Choice strategy (BC), Orthogonal Arrays (OA) and the algorithm from the Automatic Efficient Test Generator (AETG). AC satisfies n-wise coverage, EC and BC satisfy 1-wise coverage, and OA and AETG satisfy pair-wise coverage. The All Combinations strategy was used as a “gold standard” strategy; it subsumes the others but is usually too expensive for practical use. The others were used in an experiment that used five programs seeded with 128 faults. The combination strategies were evaluated with respect to the number of test cases, the number of faults found, failure size, and number of decisions covered. The strategy that requires the least number of tests, Each Choice, found the smallest number of faults. Although the Base Choice strategy requires fewer test cases than Orthogonal Arrays and AETG, it found as many faults. Analysis also shows some properties of the combination strategies that appear significant. The two most important results are that the Each Choice strategy is unpredictable in terms of which faults will be revealed, possibly indicating that faults are found by chance, and that the Base Choice and the pair-wise combination strategies to some extent target different types of faults.
|
|
| 3. |
- Grindal, Mats, et al.
(författare)
-
Combination Testing Strategies : A Survey
- 2005
-
Ingår i: Software testing, verification & reliability. - John Wiley & Sons. - 0960-0833. ; 15:3, s. 167-199
-
Tidskriftsartikel (refereegranskat)abstract
- Combination strategies are test case selection methods that identify test cases by combining values of the different test object input parameters based on some combinatorial strategy. This survey presents 16 different combination strategies, covering more than 40 papers that focus on one or several combination strategies. This collection represents most of the existing work performed on combination strategies. This survey describes the basic algorithms used by the combination strategies. Some properties of combination strategies, including coverage criteria and theoretical bounds on the size of test suites, are also included in this description. This survey paper also includes a subsumption hierarchy that attempts to relate the various coverage criteria associated with the identified combination strategies
|
|
| 4. |
- Grindal, Mats, et al.
(författare)
-
Handling Constraints in the Input Space when Using Combination Strategies for Software Testing
- 2006
-
Rapport (övrigt vetenskapligt)abstract
- This study compares seven different methods for handling constraints in input parameter models when using combination strategies to select test cases. Combination strategies are used to select test cases based on input parameter models. An input parameter model is a representation of the input space of the system under test via a set of parameters and values for these parameters. A test case is one specific combination of values for all the parameters. Sometimes the input parameter model may contain parameters that are not independent. Some sub-combinations of values of the dependent parameters may not be valid, i.e., these sub-combinations do not make sense. Combination strategies, in their basic forms, do not take into account any semantic information. Thus, invalid sub-combinations may be included in test cases in the test suite. This paper proposes four new constraint handling methods and compares these with three existing methods in an experiment in which the seven constraint handling methods are used to handle a number of different constraints in different sized input parameter models under three different coverage criteria. All in all, 2568 test suites with a total of 634,263 test cases have been generated within the scope of this experiment.
|
|
| 5. |
- Grindal, Mats, et al.
(författare)
-
Input Parameter Modeling for Combination Strategies
- 2007
-
Ingår i: Proceedings of the IASTED International Conference on Software Engineering (SE2007), Feb 13-15, Innsbruck, Austria. ; s. 255-260
-
Konferensbidrag (refereegranskat)abstract
- Combination strategies are test methods that generate test cases based on input parameter models. This paper suggests a structured modeling method used to translate requirements expressed in a general format into an input parameter model suitable for combination strategies.This paper also describes results from two initial experiments exploring the efficiency and effectiveness of the modeling method. These results indicate that the resulting models may contain enough information to detect the vast majority of faults in the system under test. Further, results indicate that the modeling method is simple enough to use in practical testing.
|
|
| 6. |
- Grindal, Mats, et al.
(författare)
-
Input Parameter Modeling for Combination Strategies in Software Testing
- 2007
-
Ingår i: Proceedings of the IASTED International Conference on Software Engineering, SE 2007. - ACTA Press. - 978-0-88986-641-6 ; s. 255-260
-
Konferensbidrag (refereegranskat)abstract
- Combination strategies are test methods that generate testcases based on input parameter models. This paper suggestsa structured modeling method used to translate requirementsexpressed in a general format into an input parametermodel suitable for combination strategies.This paper also describes results from two initial experimentsexploring the efficiency and effectiveness of themodeling method. These results indicate that the resultingmodels may contain enough information to detect the vastmajority of faults in the system under test. Further, resultsindicate that the modeling method is simple enough to usein practical testing.
|
|
| 7. |
- Grindal, Mats, et al.
(författare)
-
On the Testing Maturity of Software Producing Organizations
- 2006
-
Ingår i: Proceedings of the 1st International Conference Testing : Academia & Industry Conference Practice And Research Techniques (TAIC PART), 29th-31st August, Cumberland Lodge, Windsor, UK.
-
Konferensbidrag (refereegranskat)
|
|
| 8. |
- Grindal, Mats, et al.
(författare)
-
On the Testing Maturity of Software Producing Organizations
- 2006
-
Ingår i: Testing: Academia & Industry : Practice Ann Research Techniques (TAIC PART 2006), August 29-31, Cumberland Lodge, Windsor, UK. - IEEE Computer Society. - 0-7695-2672-1 ; s. 171-180
-
Konferensbidrag (övrigt vetenskapligt)abstract
- This paper presents data from a study of the current state of practice of software testing. Test managers from twelve different software organizations were interviewed. The interviews focused on the amount of resources spent on testing, how the testing is conducted, and the knowledge of the personnel in the test organizations. The data indicate that the overall test maturity is low. Test managers are aware of this but have trouble improving. One problem is that the organizations are commercially successful, suggesting that products must already be "good enough". Also, the current lack of structured testing in practice makes it difficult to quantify the current level of maturity and thereby articulate the potential gain from increasing testing maturity to upper management and developers
|
|
| 9. |
|
|
| 10. |
- Lindström, Birgitta, et al.
(författare)
-
Generating Trace-Sets for Model-Based Testing
- 2007
-
Ingår i: Software Reliability, 2007. ISSRE '07. The 18th IEEE International Symposium on. - IEEE Press. - 978-0-7695-3024-6 ; s. 171-180
-
Konferensbidrag (refereegranskat)abstract
- Model-checkers are powerful tools that can find individual traces through models to satisfy desired properties. These traces provide solutions to a number of problems. Instead of individual traces, software testing needs sets of traces that satisfy coverage criteria. Finding a trace set in a large model is difficult because model checkers generate single traces and use a lot of memory. Space and time requirements of modelchecking algorithms grow exponentially with respect to the number of variables and parallel automata of the model being analyzed. We present a method that generates a set of traces by iteratively invoking a model checker. The method mitigates the memory consumption problem by dynamically building partitions along the traces. This method was applied to a testability case study, and it generated the complete trace set, while ordinary model-checking could only generate 26%.
|
|
