| 1. |
- Lehman, Joel, et al.
(författare)
-
The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities
- 2020
-
Ingår i: Artificial Life. - : MIT Press - Journals. - 1530-9185 .- 1064-5462. ; 26:2, s. 274-306
-
Tidskriftsartikel (refereegranskat)abstract
- Evolution provides a creative fount of complex and subtle adaptations that often surprise the scientists who discover them. However, the creativity of evolution is not limited to the natural world: Artificial organisms evolving in computational environments have also elicited surprise and wonder from the researchers studying them. The process of evolution is an algorithmic process that transcends the substrate in which it occurs. Indeed, many researchers in the field of digital evolution can provide examples of how their evolving algorithms and organisms have creatively subverted their expectations or intentions, exposed unrecognized bugs in their code, produced unexpectedly adaptations, or engaged in behaviors and outcomes, uncannily convergent with ones found in nature. Such stories routinely reveal surprise and creativity by evolution in these digital worlds, but they rarely fit into the standard scientific narrative. Instead they are often treated as mere obstacles to be overcome, rather than results that warrant study in their own right. Bugs are fixed, experiments are refocused, and one-off surprises are collapsed into a single data point. The stories themselves are traded among researchers through oral tradition, but that mode of information transmission is inefficient and prone to error and outright loss. Moreover, the fact that these stories tend to be shared only among practitioners means that many natural scientists do not realize how interesting and lifelike digital organisms are and how natural their evolution can be. To our knowledge, no collection of such anecdotes has been published before. This article is the crowd-sourced product of researchers in the fields of artificial life and evolutionary computation who have provided first-hand accounts of such cases. It thus serves as a written, fact-checked collection of scientifically important and even entertaining stories. In doing so we also present here substantial evidence that the existence and importance of evolutionary surprises extends beyond the natural world, and may indeed be a universal property of all complex evolving systems.
|
|
| 2. |
- Alégroth, Emil, 1984, et al.
(författare)
-
Special issue on new generations of UI testing
- 2021
-
Ingår i: Software Testing Verification and Reliability. - : Wiley. - 0960-0833 .- 1099-1689. ; 31:3
-
Tidskriftsartikel (övrigt vetenskapligt/konstnärligt)
|
|
| 3. |
- Briand, Lionel, et al.
(författare)
-
JF Welcome: ICSE 2021
- 2021
-
Ingår i: Proceedings - International Conference on Software Engineering. - : IEEE Computer Society. - 0270-5257. ; May 2021, s. xxxiii-
-
Konferensbidrag (övrigt vetenskapligt/konstnärligt)
|
|
| 4. |
- Dobslaw, Felix, 1983-, et al.
(författare)
-
Automated black-box boundary value detection
- 2023
-
Ingår i: PeerJ Computer Science. - : PeerJ. - 2376-5992. ; 9
-
Tidskriftsartikel (refereegranskat)abstract
- Abstract Software systems typically have an input domain that can be subdivided into sub-domains, each of which generates similar or related outputs. Testing it on the boundaries between these sub-domains is critical to ensure high-quality software. Therefore, boundary value analysis and testing have been a fundamental part of the software testing toolbox for a long time and are typically taught early to software engineering students. Despite its many argued benefits, boundary value analysis for a given software specification or application is typically described in abstract terms. This allows for variation in how testers apply it and in the benefits they see. Additionally, its adoption has been limited since it requires a specification or model to be analysed. We propose an automated black-box boundary value detection method to support software testers in performing systematic boundary value analysis. This dynamic method can be utilized even without a specification or model. The proposed method is based on a metric referred to as the program derivative, which quantifies the level of boundariness of test inputs. By combining this metric with search algorithms, we can identify and rank pairs of inputs as good boundary candidates, i.e., inputs that are in close proximity to each other but with outputs that are far apart. We have implemented the AutoBVA approach and evaluated it on a curated dataset of example programs. Furthermore, we have applied the approach broadly to a sample of 613 functions from the base library of the Julia programming language. The approach could identify boundary candidates that highlight diverse boundary behaviours in over 70% of investigated systems under test. The results demonstrate that even a simple variant of the program derivative, combined with broad sampling and search over the input space, can identify interesting boundary candidates for a significant portion of the functions under investigation. In conclusion, we also discuss the future extension of the approach to encompass more complex systems under test cases and datatypes.
|
|
| 5. |
- Dobslaw, Felix, et al.
(författare)
-
Automated black-box boundary value detection
- 2023
-
Ingår i: PeerJ Computer Science. - 2376-5992. ; 9
-
Tidskriftsartikel (refereegranskat)abstract
- Software systems typically have an input domain that can be subdivided into subdomains, each of which generates similar or related outputs. Testing it on the boundaries between these sub-domains is critical to ensure high-quality software. Therefore, boundary value analysis and testing have been a fundamental part of the software testing toolbox for a long time and are typically taught early to software engineering students. Despite its many argued benefits, boundary value analysis for a given software specification or application is typically described in abstract terms. This allows for variation in how testers apply it and in the benefits they see. Additionally, its adoption has been limited since it requires a specification or model to be analysed. We propose an automated black-box boundary value detection method to support software testers in performing systematic boundary value analysis. This dynamic method can be utilized even without a specification or model. The proposed method is based on a metric referred to as the program derivative, which quantifies the level of boundariness of test inputs. By combining this metric with search algorithms, we can identify and rank pairs of inputs as good boundary candidates, i.e., inputs that are in close proximity to each other but with outputs that are far apart. We have implemented the AutoBVA approach and evaluated it on a curated dataset of example programs. Furthermore, we have applied the approach broadly to a sample of 613 functions from the base library of the Julia programming language. The approach could identify boundary candidates that highlight diverse boundary behaviours in over 70% of investigated systems under test. The results demonstrate that even a simple variant of the program derivative, combined with broad sampling and search over the input space, can identify interesting boundary candidates for a significant portion of the functions under investigation. In conclusion, we also discuss the future extension of the approach to encompass more complex systems under test cases and datatypes.
|
|
| 6. |
- Dobslaw, Felix, et al.
(författare)
-
Automated black-box boundary value detection
- 2023
-
Ingår i: PeerJ Computer Science. - 2376-5992. ; 9
-
Tidskriftsartikel (refereegranskat)abstract
- Software systems typically have an input domain that can be subdivided into subdomains, each of which generates similar or related outputs. Testing it on the boundaries between these sub-domains is critical to ensure high-quality software. Therefore, boundary value analysis and testing have been a fundamental part of the software testing toolbox for a long time and are typically taught early to software engineering students. Despite its many argued benefits, boundary value analysis for a given software specification or application is typically described in abstract terms. This allows for variation in how testers apply it and in the benefits they see. Additionally, its adoption has been limited since it requires a specification or model to be analysed. We propose an automated black-box boundary value detection method to support software testers in performing systematic boundary value analysis. This dynamic method can be utilized even without a specification or model. The proposed method is based on a metric referred to as the program derivative, which quantifies the level of boundariness of test inputs. By combining this metric with search algorithms, we can identify and rank pairs of inputs as good boundary candidates, i.e., inputs that are in close proximity to each other but with outputs that are far apart. We have implemented the AutoBVA approach and evaluated it on a curated dataset of example programs. Furthermore, we have applied the approach broadly to a sample of 613 functions from the base library of the Julia programming language. The approach could identify boundary candidates that highlight diverse boundary behaviours in over 70% of investigated systems under test. The results demonstrate that even a simple variant of the program derivative, combined with broad sampling and search over the input space, can identify interesting boundary candidates for a significant portion of the functions under investigation. In conclusion, we also discuss the future extension of the approach to encompass more complex systems under test cases and datatypes.
|
|
| 7. |
- Dobslaw, Felix, 1983, et al.
(författare)
-
Boundary Value Exploration for Software Analysis
- 2020
-
Ingår i: Proceedings - 2020 IEEE 13th International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2020. - : IEEE. ; , s. 346-353
-
Konferensbidrag (refereegranskat)abstract
- For software to be reliable and resilient, it is widely accepted that tests must be created and maintained alongside the software itself. One safeguard from vulnerabilities and failures in code is to ensure correct behavior on the boundaries between subdomains of the input space. So-called boundary value analysis (BVA) and boundary value testing (BVT) techniques aim to exercise those boundaries and increase test effectiveness. However, the concepts of BVA and BVT themselves are not generally well defined, and it is not clear how to identify relevant sub-domains, and thus the boundaries delineating them, given a specification. This has limited adoption and hindered automation. We clarify BVA and BVT and introduce Boundary Value Exploration (BVE) to describe techniques that support them by helping to detect and identify boundary inputs. Additionally, we propose two concrete BVE techniques based on information-theoretic distance functions: (i) an algorithm for boundary detection and (ii) the usage of software visualization to explore the behavior of the software under test and identify its boundary behavior. As an initial evaluation, we apply these techniques on a much used and well-tested date handling library. Our results reveal questionable behavior at boundaries highlighted by our techniques. In conclusion, we argue that the boundary value exploration that our techniques enable is a step towards automated boundary value analysis and testing, which can foster their wider use and improve test effectiveness and efficiency.
|
|
| 8. |
- Dobslaw, Felix, 1983-, et al.
(författare)
-
Similarities of Testing Programmed and Learnt Software
- 2023
-
Ingår i: Proceedings - 2023 IEEE 16th International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2023. - : IEEE conference proceedings. ; , s. 78-81
-
Konferensbidrag (refereegranskat)abstract
- This study examines to what extent the testing of traditional software components and machine learning (ML) models fundamentally differs or not. While some researchers argue that ML software requires new concepts and perspectives for testing, our analysis highlights that, at a fundamental level, the specification and testing of a software component are not dependent on the development process used or on implementation details. Although the software engineering/computer science (SE/CS) and Data Science/ML (DS/ML) communities have developed different expectations, unique perspectives, and varying testing methods, they share clear commonalities that can be leveraged. We argue that both areas can learn from each other, and a non-dual perspective could provide novel insights not only for testing ML but also for testing traditional software. Therefore, we call upon researchers from both communities to collaborate more closely and develop testing methods and tools that can address both traditional and ML software components. While acknowledging their differences has merits, we believe there is great potential in working on unified methods and tools that can address both types of software.
|
|
| 9. |
- Enoiu, Eduard Paul, PhD, et al.
(författare)
-
Towards a Model of Testers' Cognitive Processes: Software Testing as a Problem Solving Approach
- 2020
-
Ingår i: Proceedings - Companion of the 2020 IEEE 20th International Conference on Software Quality, Reliability, and Security, QRS-C 2020. ; , s. 272-279
-
Konferensbidrag (refereegranskat)abstract
- Software testing is a complex, intellectual activity based (at least) on analysis, reasoning, decision making, abstraction and collaboration performed in a highly demanding environment. Naturally, it uses and allocates multiple cognitive resources in software testers. However, while a cognitive psychology perspective is increasingly used in the general software engineering literature, it has yet to find its place in software testing. To the best of our knowledge, no theory of software testers' cognitive processes exists. Here, we take the first step towards such a theory by presenting a cognitive model of software testing based on how problem solving is conceptualized in cognitive psychology. Our approach is to instantiate a general problem solving process for the specific problem of creating test cases. We then propose an experiment for testing our cognitive test design model. The experiment makes use of verbal protocol analysis to understand the mechanisms by which human testers choose, design, implement and evaluate test cases. An initial evaluation was then performed with five software engineering master students as subjects. The results support a problem solving-based model of test design for capturing testers' cognitive processes.
|
|
| 10. |
- Enoiu, Eduard Paul, PhD, et al.
(författare)
-
Towards Human-Like Automated Test Generation: Perspectives from Cognition and Problem Solving
- 2021
-
Ingår i: Proceedings - 2021 IEEE/ACM 13th International Workshop on Cooperative and Human Aspects of Software Engineering, CHASE 2021. - Virtual and Madrid, Spain. ; , s. 123-124
-
Konferensbidrag (refereegranskat)abstract
- Automated testing tools typically create test cases that are different from what human testers create. This often makes the tools less effective, the created tests harder to understand, and thus results in tools providing less support to human testers. Here, we propose a framework based on cognitive science and, in particular, an analysis of approaches to problem solving, for identifying cognitive processes of testers. The framework helps map test design steps and criteria used in human test activities and thus to better understand how effective human testers perform their tasks. Ultimately, our goal is to be able to mimic how humans create test cases and thus to design more human-like automated test generation systems. We posit that such systems can better augment and support testers in a way that is meaningful to them.
|
|
