| 1. |
- Bisht, Rohini, et al.
(författare)
-
Identifying Redundancies and Gaps Across Testing Levels During Verification of Automotive Software
- 2023
-
Ingår i: 2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). - 2159-4848. - 9798350333350 ; , s. 131-139
-
Konferensbidrag (refereegranskat)abstract
- Testing of automotive systems usually follows the V-Model, a process where sequential testing activities progress from low-level code structures to high-level integrated systems. In theory, the V-Model should reduce redundant testing and prevent gaps in verification. To assess whether such benefits translate in practice, in a case study at Scania CV AB, we have developed a framework to identify redundancies and gaps in test cases across V-model test levels.Our framework identified both redundancies and gaps in Sca-nia’s scripted testing efforts. Deviating cases were also identified where, e.g., requirements were outdated or contained incorrect details. Factors contributing to redundancy include re-verification in a new context, difficulties mapping requirements across levels, and lack of test case documentation. Both redundancies and gaps result from a lack of communication and traceability of test results across test levels. We recommend active collaboration across levels, as well as use of coverage matrices to alleviate these issues. We offer our framework to help refine testing practices and to inspire process improvements.
|
|
| 2. |
- Alégroth, Emil, 1984, et al.
(författare)
-
Conceptualization and evaluation of component-based testing unified with visual GUI testing: An empirical study
- 2015
-
Ingår i: 2015 IEEE 8th International Conference on Software Testing, Verification and Validation, ICST 2015 - Proceedings. - 2159-4848. - 9781479971251
-
Konferensbidrag (refereegranskat)abstract
- In this paper we present the results of a two-phase empirical study where we evaluate and compare the applicability of automated component-based Graphical User Interface (GUI) testing and Visual GUI Testing (VGT) in the tools GUITAR and a prototype tool we refer to as VGT GUITAR. First, GUI mutation operators are defined to create 18 faulty versions of an application on which both tools are then applied in an experiment. Results from 456 test case executions in each tool show, with statistical significance, that the component-based approach reports more false negatives than VGT for acceptance tests but that the VGT approach reports more false positives for system tests. Second, a case study is performed with larger open source applications, ranging from 8,803-55,006 lines of code. Results show that GUITAR is applicable in practice but has some challenges related to GUI component states. The results also show that VGT GUITAR is currently not applicable in practice and therefore requires further research and development. Based on the study's results we present areas of future work for both test approaches and conclude that the approaches have different benefits and drawbacks. The component-based approach is robust and executes tests faster than the VGT approach, with a factor of 3. However, the VGT approach can perform visual assertions and is perceived more flexible than the component- based approach. These conclusions let us hypothesize that a combination of the two approaches is the most suitable in practice and therefore warrants future research.
|
|
| 3. |
- Alégroth, Emil, 1984, et al.
(författare)
-
Exploring the Presence of Technical Debt in Industrial GUI-based Testware: A Case Study
- 2016
-
Ingår i: 2016 IEEE Ninth International Conference on Software Testing, Verification and Validation Workshops. - : IEEE. - 2159-4848. - 9781509036745 ; , s. 257-262
-
Konferensbidrag (refereegranskat)abstract
- Technical debt (TD) is a concept used to describe a sub-optimal solution of a software artifact that negatively affects its comprehensibility, extendability and maintainability. As such, TD adversely affects the costs or quality associated with the artifact, which is also called interest. TD has through research been identified in all types of software artifacts, from architectural design to automated tests (Testware). However, research into testware technical debt (TTD) is limited and primarily focused on testing on lower level of system abstraction, i.e. unit-and integration tests, leaving a need for more TTD research on GUI-based testing. In this study we explore this gap in knowledge through an industrial case study at a Swedish avionics software development company. Four repositories are studied for the presence of TTD using expert interviews, semi-automated document analysis and automatic metric analysis. Results of the study provide initial support that the concept of TTD is applicable to GUI-based testware and show the presence of both TD items unique to GUI-based testware and items common to software. The implications of these results are that engineering best practices must be established for GUI-based testware to minimize TD interest.
|
|
| 4. |
- Alégroth, Emil, 1984, et al.
(författare)
-
On the industrial applicability of TextTest: An empirical case study
- 2015
-
Ingår i: 2015 IEEE 8th International Conference on Software Testing, Verification and Validation, ICST 2015 - Proceedings. - 2159-4848. - 9781479971251
-
Konferensbidrag (refereegranskat)abstract
- Software systems are becoming more complex, not least in their Graphical User Interfaces (GUIs), which presents challenges for existing testing practices. Pressure to reduce time to market leaves less time for manual testing and increases the importance of test automation. Previous research has identified several generations of automated GUI-based test approaches with different cost-benefit tradeoffs. Whilst test automation provides fast quality feedback it can be associated with high costs and inability to identify defects not explicitly anticipated by the test designer. TextTest is a capture-replay tool for GUI-based testing with a novel approach that overcomes several of the challenges experienced with previous approaches. Firstly the tool supports Approval Testing, an approach where ASCII-art representations of the GUI's visual state are used to verify correct application behavior at the system level. Secondly it records and replays test scripts in a user defined domain specific language (DSL) that is readable by all stakeholders. In this paper we present a three phase industrial case study that aims to identify TextTest's applicability in industrial practice. The paper reports that the tool is associated with (1) low script development costs due to recording functionality, (2) low maintenance costs, on average 7 minutes per test case, (3) better defect finding ability than manual system testing, (4) high test case execution performance (In this case 500 test cases in 20 minutes), (5) high script readability due to DSL defined scripts, and (6) test suites that are robust to change (In this case 93 percent per iteration). However, the tool requires a higher degree of technical skill for customization work, test maintainers need skills in designing regular expressions and the tool's applicability is currently restricted to Java and Python based applications.
|
|
| 5. |
- Arts, Thomas, 1969, et al.
(författare)
-
Graphical editing support for QuickCheck models
- 2015
-
Ingår i: 2015 IEEE 8th International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2015 - Proceedings. - 2159-4848. - 9781479918850
-
Konferensbidrag (refereegranskat)abstract
- QuickCheck can test a system by specifying a state machine for the API exported by that system. This state machine specification includes a list of possible API calls. Each call is accompanied by a precondition, a postcondition, a generator for the arguments, and a description of how the state is changed. Based on this specification QuickCheck generates a random sequence of API calls. The preconditions ensure that a generated sequence is valid, and the postconditions check that the system behaves as expected. Many systems require an initialisation call before other calls, describing the transition from an uninitialized to an initialised state. QuickCheck offers two ways of specifying transitions between states: using preconditions or a finite state machine abstraction. In this paper we show, by means of an example, that the latter approach is to be preferred. In addition, we present a recent extension to QuickCheck that allows a user to graphically create and edit a finite state machine specification. This extension simplifies and speeds up the specification of a finite state machine, which can be regarded as a formal model of the system, considerably. The graphical representation makes the formal model easier to understand, and visualises the distribution of API calls. Moreover, the extension allows the user to change this distribution.
|
|
| 6. |
- Arts, Thomas, 1969, et al.
(författare)
-
Testing AUTOSAR software with QuickCheck
- 2015
-
Ingår i: 2015 IEEE 8th International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2015 - Proceedings. - 2159-4848. - 9781479918850
-
Konferensbidrag (refereegranskat)abstract
- AUTOSAR (AUTomotive Open System ARchitecture) is an evolving standard for embedded software in vehicles, defined by the automotive industry, and implemented by many different vendors. On behalf of Volvo Cars, we have developed model-based acceptance tests for some critical AUTOSAR components, to guarantee that implementations from different vendors are compatible. We translated over 3000 pages of textual specifications into QuickCheck models, and tested many different implementations using large volumes of generated tests. This exposed over 200 issues, which we raised with Volvo and the software vendors. Compared to an earlier manual approach, ours is more efficient, more effective, and more correct.
|
|
| 7. |
- Barrett, Ayodele, et al.
(författare)
-
On the Current State of Academic Software Testing Education in Sweden
- 2023
-
Ingår i: Proceedings - 2023 IEEE 16th International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2023. - : Institute of Electrical and Electronics Engineers Inc.. - 9798350333350 ; , s. 397-404
-
Konferensbidrag (refereegranskat)abstract
- Well-trained software development personnel, in the art and science of software testing, will effectively and efficiently develop quality software products with potentially fewer, less-critical defects. Thus software testing education is considered to be an important part of curricula for a university degree in Computer Science or Information Systems. The objective of this paper is to determine how much dedicated knowledge in the field of software testing is taught within Swedish universities. To achieve this objective, a systematic search of syllabi for software testing-related courses was done. From 25 Swedish universities offering Computer Science (or related) degrees, 14 currently offer dedicated courses in software testing. Some findings include: 32% of the individual courses were offered at the undergraduate level; 28% of the universities offer courses for specialised testing training; and, for the vast majority of the universities, dedicated software testing courses account for about 5% of the total degree credits. While some universities fare better than others, the overall state of academic software testing education in Sweden is limited but promising.
|
|
| 8. |
- Bauer, Andreas, et al.
(författare)
-
We Tried and Failed : An Experience Report on a Collaborative Workflow for GUI-based Testing
- 2023
-
Ingår i: Proceedings - 2023 IEEE 16th International Conference on Software Testing, Verification and Validation Workshops, ICSTW. - : Institute of Electrical and Electronics Engineers (IEEE). - 9798350333350 ; , s. 1-9
-
Konferensbidrag (refereegranskat)abstract
- Modern software development is a team-based effort supported by tools, processes, and practices. One integral part is automated testing, where developers incorporate automated tests on multiple levels of system abstraction, from low-level unit tests to high-level system tests and Graphical User Interface (GUI) tests. Furthermore, the common practices of code reviews allow collaboration on artifacts based on discussions that improve the artifact's quality and to share information within the team. However, the characteristics of GUI-based tests, due to the level of abstraction and visual elements, introduce additional requirements and complexities compared to code or lower-level test code review, delimiting the practice benefits.The objective of this work is to propose a tool-supported workflow that enables active collaboration among stakeholders and improves the efficiency and effectiveness of team-based development of GUI-based tests.To evaluate the workflow, and show proof of concept, a technical demonstrator for merging of GUI-based tests was to be developed. However, during its development, we encountered several unforeseen challenges that forced us to halt its development. We report the negative results from this development and the main challenges we encountered, as well as the rationale and the decisions we took towards this workflow.In conclusion, this work presents a negative research result on a failed attempt to propose a tool-supported workflow that enables active collaboration on GUI-based tests. The outcome and learnings of this work are intended to guide future research and prevent researchers from falling into the same pitfalls we did. © 2023 IEEE.
|
|
| 9. |
- Berglund, Lukas, et al.
(författare)
-
Test Maintenance for Machine Learning Systems: A Case Study in the Automotive Industry
- 2023
-
Ingår i: 2023 IEEE Conference on Software Testing, Verification and Validation (ICST). - 2159-4848. - 9781665456661 ; , s. 410-421
-
Konferensbidrag (refereegranskat)abstract
- Machine Learning (ML) systems have seen widespread use for automated decision making. Testing is essential to ensure the quality of these systems, especially safety-critical autonomous systems in the automotive domain. ML systems introduce new challenges with the potential to affect test maintenance, the process of updating test cases to match the evolving system. We conducted an exploratory case study in the automotive domain to identify factors that affect test maintenance for ML systems, as well as to make recommendations to improve the maintenance process. Based on interview and artifact analysis, we identified 14 factors affecting maintenance, including five especially relevant for ML systems—with the most important relating to non-determinism and large input spaces. We also proposed ten recommendations for improving test maintenance, including four targeting ML systems—in particular, emphasizing the use of test oracles tolerant to acceptable non-determinism. The study’s findings expand our knowledge of test maintenance for an emerging class of systems, benefiting the practitioners testing these systems.
|
|
| 10. |
- Gay, Gregory, 1987, et al.
(författare)
-
How Closely are Common Mutation Operators Coupled to Real Faults?
- 2023
-
Ingår i: Proceedings - 2023 IEEE 16th International Conference on Software Testing, Verification and Validation, ICST 2023. ; , s. 129-140
-
Konferensbidrag (refereegranskat)abstract
- In mutation testing, faulty versions of a program are generated through automated modifications of source code. These mutants are used to assess and improve test suite quality, under the assumption that detection of mutants is indicative of a test suite's ability to detect real faults - i.e., that mutants and faults have a semantic relationship. Improving the effectiveness - in both cost and quality - of mutation testing may lie in better understanding this relationship, in particular with regard to how individual mutation operators (types) couple to real faults.In this study, we examine coupling between 32,002 mutants produced by 31 mutation operators and 144 real faults, using a scale based on number of failing tests and reasons for failure. Ultimately, we observed that 9.92% of the mutants are strongly coupled to real faults, and 51.03% of the faults have at least one strongly coupled mutant. We identify and examine mutation operators with the highest median coupling, as well as the operators that tend to produce non-compiling mutants, undetected mutants, and mutants that cause tests other than those that detect the actual fault to fail. We also examine how coupling could be used to filter the set of operators employed, leading to potentially significant cost savings during mutation testing. Our findings could lead to improvements in how mutation testing is applied, improved implementation of specific mutation operators, and inspiration for new mutation operators.
|
|
