| 1. |
- Almulla, Hussein, et al.
(författare)
-
Generating Diverse Test Suites for Gson Through Adaptive Fitness Function Selection
- 2020
-
Ingår i: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). - Cham : Springer International Publishing. - 1611-3349 .- 0302-9743. - 9783030597610 - 9783030597627 ; SSBSE 2020, s. 246-252
-
Konferensbidrag (refereegranskat)abstract
- Many fitness functions - such as those targeting test suite diversity—do not yield sufficient feedback to drive test generation. We propose that diversity can instead be improved through adaptive fitness function selection (AFFS), an approach that varies the fitness functions used throughout the generation process in order to strategically increase diversity. We have evaluated our AFFS framework, EvoSuiteFIT, on a set of 18 real faults from Gson, a JSON (de)serialization library. Ultimately, we find that AFFS creates test suites that are more diverse than those created using static fitness functions. We also observe that increased diversity may lead to small improvements in the likelihood of fault detection.
|
|
| 2. |
- Almulla, H., et al.
(författare)
-
Learning how to search: generating effective test cases through adaptive fitness function selection
- 2022
-
Ingår i: Empirical Software Engineering. - : Springer Science and Business Media LLC. - 1382-3256 .- 1573-7616. ; 27:2
-
Tidskriftsartikel (refereegranskat)abstract
- Search-based test generation is guided by feedback from one or more fitness functions-scoring functions that judge solution optimality. Choosing informative fitness functions is crucial to meeting the goals of a tester. Unfortunately, many goals-such as forcing the class-under-test to throw exceptions, increasing test suite diversity, and attaining Strong Mutation Coverage-do not have effective fitness function formulations. We propose that meeting such goals requires treating fitness function identification as a secondary optimization step. An adaptive algorithm that can vary the selection of fitness functions could adjust its selection throughout the generation process to maximize goal attainment, based on the current population of test suites. To test this hypothesis, we have implemented two reinforcement learning algorithms in the EvoSuite unit test generation framework, and used these algorithms to dynamically set the fitness functions used during generation for the three goals identified above. We have evaluated our framework, EvoSuiteFIT, on a set of Java case examples. EvoSuiteFIT techniques attain significant improvements for two of the three goals, and show limited improvements on the third when the number of generations of evolution is fixed. Additionally, for two of the three goals, EvoSuiteFIT detects faults missed by the other techniques. The ability to adjust fitness functions allows strategic choices that efficiently produce more effective test suites, and examining these choices offers insight into how to attain our testing goals. We find that adaptive fitness function selection is a powerful technique to apply when an effective fitness function does not already exist for achieving a testing goal.
|
|
| 3. |
- Almulla, Hussein, et al.
(författare)
-
Learning How to Search: Generating Exception-Triggering Tests Through Adaptive Fitness Function Selection
- 2020
-
Ingår i: Proceedings - 2020 IEEE 13th International Conference on Software Testing, Verification and Validation, ICST 2020. - Porto, Portugal : IEEE. ; , s. 63-73
-
Konferensbidrag (refereegranskat)abstract
- Search-based test generation is guided by feedback from one or more fitness functions—scoring functions that judge solution optimality. Choosing informative fitness functions is crucial to meeting the goals of a tester. Unfortunately, many goals—such as forcing the class-under-test to throw exceptions— do not have a known fitness function formulation. We propose that meeting such goals requires treating fitness function identification as a secondary optimization step. An adaptive algorithm that can vary the selection of fitness functions could adjust its selection throughout the generation process to maximize goal attainment, based on the current population of test suites. To test this hypothesis, we have implemented two reinforcement learning algorithms in the EvoSuite framework, and used these algorithms to dynamically set the fitness functions used during generation.We have evaluated our framework, EvoSuiteFIT, on a set of 386 real faults. EvoSuiteFIT discovers and retains more exception-triggering input and produces suites that detect a variety of faults missed by the other techniques. The ability to adjust fitness functions allows EvoSuiteFIT to make strategic choices that efficiently produce more effective test suites.
|
|
| 4. |
- Bauer, Andreas
(författare)
-
Towards Collaborative GUI-based Testing
- 2023
-
Licentiatavhandling (övrigt vetenskapligt/konstnärligt)abstract
- Context:Contemporary software development is a socio-technical activity requiring extensive collaboration among individuals with diverse expertise.Software testing is an integral part of software development that also depends on various expertise.GUI-based testing allows assessing a system’s GUI and its behavior through its graphical user interface.Collaborative practices in software development, like code reviews, not only improve software quality but also promote knowledge exchange within teams.Similar benefits could be extended to other areas of software engineering, such as GUI-based testing.However, collaborative practices for GUI-based testing necessitate a unique approach since general software development practices, perceivably, can not be directly transferred to software testing.Goal:This thesis contributes towards a tool-supported approach enabling collaborative GUI-based testing.Our distinct goals are (1) to identify processes and guidelines to enable collaboration on GUI-based testing artifacts and (2) to operationalize tool support to aid this collaboration.Method:We conducted a systematic literature review identifying code review guidelines for GUI-based testing.Further, we conducted a controlled experiment to assess the efficiency and potential usability issues of Augmented Testing.Results:We provided guidelines for reviewing GUI-based testing artifacts, which aid contributors and reviewers during code reviews.We further provide empirical evidence that Augmented Testing is not only an efficient approach to GUI-based testing but also usable for non-technical users, making it a promising subject for further research in collaborative GUI-based testing.Conclusion:Code review guidelines aid collaboration through discussions, and a suitable testing approach can serve as a platform to operationalize collaboration.Collaborative GUI-based testing has the potential to improve the efficiency and effectiveness of such testing.
|
|
| 5. |
- Berglund, Lukas, et al.
(författare)
-
Test Maintenance for Machine Learning Systems: A Case Study in the Automotive Industry
- 2023
-
Ingår i: 2023 IEEE Conference on Software Testing, Verification and Validation (ICST). - 2159-4848. - 9781665456661 ; , s. 410-421
-
Konferensbidrag (refereegranskat)abstract
- Machine Learning (ML) systems have seen widespread use for automated decision making. Testing is essential to ensure the quality of these systems, especially safety-critical autonomous systems in the automotive domain. ML systems introduce new challenges with the potential to affect test maintenance, the process of updating test cases to match the evolving system. We conducted an exploratory case study in the automotive domain to identify factors that affect test maintenance for ML systems, as well as to make recommendations to improve the maintenance process. Based on interview and artifact analysis, we identified 14 factors affecting maintenance, including five especially relevant for ML systems—with the most important relating to non-determinism and large input spaces. We also proposed ten recommendations for improving test maintenance, including four targeting ML systems—in particular, emphasizing the use of test oracles tolerant to acceptable non-determinism. The study’s findings expand our knowledge of test maintenance for an emerging class of systems, benefiting the practitioners testing these systems.
|
|
| 6. |
- Bisht, Rohini, et al.
(författare)
-
Identifying Redundancies and Gaps Across Testing Levels During Verification of Automotive Software
- 2023
-
Ingår i: 2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). - 2159-4848. - 9798350333350 ; , s. 131-139
-
Konferensbidrag (refereegranskat)abstract
- Testing of automotive systems usually follows the V-Model, a process where sequential testing activities progress from low-level code structures to high-level integrated systems. In theory, the V-Model should reduce redundant testing and prevent gaps in verification. To assess whether such benefits translate in practice, in a case study at Scania CV AB, we have developed a framework to identify redundancies and gaps in test cases across V-model test levels.Our framework identified both redundancies and gaps in Sca-nia’s scripted testing efforts. Deviating cases were also identified where, e.g., requirements were outdated or contained incorrect details. Factors contributing to redundancy include re-verification in a new context, difficulties mapping requirements across levels, and lack of test case documentation. Both redundancies and gaps result from a lack of communication and traceability of test results across test levels. We recommend active collaboration across levels, as well as use of coverage matrices to alleviate these issues. We offer our framework to help refine testing practices and to inspire process improvements.
|
|
| 7. |
- Bollina, Srujana, et al.
(författare)
-
Bytecode-Based Multiple Condition Coverage: An Initial Investigation
- 2020
-
Ingår i: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). - Cham : Springer International Publishing. - 1611-3349 .- 0302-9743. - 9783030597627 ; SSBSE 2020, s. 220-236
-
Konferensbidrag (refereegranskat)abstract
- Masking occurs when one condition prevents another from influencing the output of a Boolean expression. Adequacy criteria such as Multiple Condition Coverage (MCC) overcome masking within one expression, but offer no guarantees about subsequent expressions. As a result, a Boolean expression written as a single complex statement will yield more effective test cases than when written as a series of simple expressions. Many approaches to automated test case generation for Java operate not on the source code, but on bytecode. The transformation to bytecode simplifies complex expressions into multiple expressions, introducing masking. We propose Bytecode-MCC, a new adequacy criterion designed to group bytecode expressions and reformulate them into complex expressions. Bytecode-MCC should produce test obligations that are more likely to reveal faults in program logic than tests covering the simplified bytecode. A preliminary study shows potential improvements from attaining Bytecode-MCC coverage. However, Bytecode-MCC is difficult to optimize, and means of increasing coverage are needed before the technique can make a difference in practice. We propose potential methods to improve coverage.
|
|
| 8. |
- Borg, Markus, et al.
(författare)
-
Summary of the 4th International Workshop on Requirements Engineering and Testing (RET 2017)
- 2018
-
Ingår i: Software Engineering Notes. - : Association for Computing Machinery (ACM). - 0163-5948 .- 1943-5843. ; 42:4, s. 28-31
-
Tidskriftsartikel (övrigt vetenskapligt/konstnärligt)abstract
- The RET (Requirements Engineering and Testing) workshop series provides a meeting point for researchers and practitioners from the two separate fields of Requirements Engineering (RE) and Testing. The long term aim is to build a community and a body of knowledge within the intersection of RE and Testing, i.e., RET. The 4th workshop was co-located with the 25th International Requirements Engineering Conference (RE'17) in Lisbon, Portugal and attracted about 20 participants. In line with the previous workshop instances, RET 2017 o ered an interactive setting with a keynote, an invited talk, paper presentations, and a concluding hands-on exercise.
|
|
| 9. |
- Ebadi, Hamid, et al.
(författare)
-
Efficient and Effective Generation of Test Cases for Pedestrian Detection - Search-based Software Testing of Baidu Apollo in SVL
- 2021
-
Ingår i: 2021 IEEE International Conference on Artificial Intelligence Testing (AITest). - : IEEE. - 9781665434812 ; , s. 103-110
-
Konferensbidrag (refereegranskat)abstract
- With the growing capabilities of autonomous vehicles, there is a higher demand for sophisticated and pragmatic quality assurance approaches for machine learning-enabled systems in the automotive AI context. The use of simulation-based prototyping platforms provides the possibility for early-stage testing, enabling inexpensive testing and the ability to capture critical corner-case test scenarios. Simulation-based testing properly complements conventional on-road testing. However, due to the large space of test input parameters in these systems, the efficient generation of effective test scenarios leading to the unveiling of failures is a challenge. This paper presents a study on testing pedestrian detection and emergency braking system of the Baidu Apollo autonomous driving platform within the SVL simulator. We propose an evolutionary automated test generation technique that generates failure-revealing scenarios for Apollo in the SVL environment. Our approach models the input space using a generic and flexible data structure and benefits a multi-criteria safety-based heuristic for the objective function targeted for optimization. This paper presents the results of our proposed test generation technique in the 2021 IEEE Autonomous Driving AI Test Challenge. In order to demonstrate the efficiency and effectiveness of our approach, we also report the results from a baseline random generation technique. Our evaluation shows that the proposed evolutionary test case generator is more effective at generating failure-revealing test cases and provides higher diversity between the generated failures than the random baseline.
|
|
| 10. |
- Enoiu, Eduard Paul, PhD, et al.
(författare)
-
Understanding Problem Solving in Software Testing : An Exploration of Tester Routines and Behavior
- 2023
-
Ingår i: Lecture Notes Computer Science. - : Springer Science and Business Media Deutschland GmbH. - 9783031432392 ; 14131 LNCS, s. 143-159
-
Konferensbidrag (refereegranskat)abstract
- Software testing is a difficult, intellectual activity performed in a social environment. Naturally, testers use and allocate multiple cognitive resources towards this task. The goal of this study is to understand better the routine and behaviour of human testers and their mental models when performing testing. We investigate this topic by surveying 38 software testers and developers in Sweden. The survey explores testers’ cognitive processes when performing testing by investigating the knowledge they bring, the activities they select and perform, and the challenges they face in their routine. By analyzing the survey results, we provide a characterization of tester practices and identify insights regarding the problem-solving process. We use these descriptions to further enhance a cognitive model of software testing. © 2023, IFIP International Federation for Information Processing.
|
|
