| 1. |
- Barrett, Ayodele, et al.
(författare)
-
On the Current State of Academic Software Testing Education in Sweden
- 2023
-
Ingår i: Proceedings - 2023 IEEE 16th International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2023. - : Institute of Electrical and Electronics Engineers Inc.. - 9798350333350 ; , s. 397-404
-
Konferensbidrag (refereegranskat)abstract
- Well-trained software development personnel, in the art and science of software testing, will effectively and efficiently develop quality software products with potentially fewer, less-critical defects. Thus software testing education is considered to be an important part of curricula for a university degree in Computer Science or Information Systems. The objective of this paper is to determine how much dedicated knowledge in the field of software testing is taught within Swedish universities. To achieve this objective, a systematic search of syllabi for software testing-related courses was done. From 25 Swedish universities offering Computer Science (or related) degrees, 14 currently offer dedicated courses in software testing. Some findings include: 32% of the individual courses were offered at the undergraduate level; 28% of the universities offer courses for specialised testing training; and, for the vast majority of the universities, dedicated software testing courses account for about 5% of the total degree credits. While some universities fare better than others, the overall state of academic software testing education in Sweden is limited but promising.
|
|
| 2. |
- Bauer, Andreas, et al.
(författare)
-
We Tried and Failed : An Experience Report on a Collaborative Workflow for GUI-based Testing
- 2023
-
Ingår i: Proceedings - 2023 IEEE 16th International Conference on Software Testing, Verification and Validation Workshops, ICSTW. - : Institute of Electrical and Electronics Engineers (IEEE). - 9798350333350 ; , s. 1-9
-
Konferensbidrag (refereegranskat)abstract
- Modern software development is a team-based effort supported by tools, processes, and practices. One integral part is automated testing, where developers incorporate automated tests on multiple levels of system abstraction, from low-level unit tests to high-level system tests and Graphical User Interface (GUI) tests. Furthermore, the common practices of code reviews allow collaboration on artifacts based on discussions that improve the artifact's quality and to share information within the team. However, the characteristics of GUI-based tests, due to the level of abstraction and visual elements, introduce additional requirements and complexities compared to code or lower-level test code review, delimiting the practice benefits.The objective of this work is to propose a tool-supported workflow that enables active collaboration among stakeholders and improves the efficiency and effectiveness of team-based development of GUI-based tests.To evaluate the workflow, and show proof of concept, a technical demonstrator for merging of GUI-based tests was to be developed. However, during its development, we encountered several unforeseen challenges that forced us to halt its development. We report the negative results from this development and the main challenges we encountered, as well as the rationale and the decisions we took towards this workflow.In conclusion, this work presents a negative research result on a failed attempt to propose a tool-supported workflow that enables active collaboration on GUI-based tests. The outcome and learnings of this work are intended to guide future research and prevent researchers from falling into the same pitfalls we did. © 2023 IEEE.
|
|
| 3. |
- Bisht, Rohini, et al.
(författare)
-
Identifying Redundancies and Gaps Across Testing Levels During Verification of Automotive Software
- 2023
-
Ingår i: 2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). - 2159-4848. - 9798350333350 ; , s. 131-139
-
Konferensbidrag (refereegranskat)abstract
- Testing of automotive systems usually follows the V-Model, a process where sequential testing activities progress from low-level code structures to high-level integrated systems. In theory, the V-Model should reduce redundant testing and prevent gaps in verification. To assess whether such benefits translate in practice, in a case study at Scania CV AB, we have developed a framework to identify redundancies and gaps in test cases across V-model test levels.Our framework identified both redundancies and gaps in Sca-nia’s scripted testing efforts. Deviating cases were also identified where, e.g., requirements were outdated or contained incorrect details. Factors contributing to redundancy include re-verification in a new context, difficulties mapping requirements across levels, and lack of test case documentation. Both redundancies and gaps result from a lack of communication and traceability of test results across test levels. We recommend active collaboration across levels, as well as use of coverage matrices to alleviate these issues. We offer our framework to help refine testing practices and to inspire process improvements.
|
|
| 4. |
- Gu, Rong, 1987-, et al.
(författare)
-
Model-Based Policy Synthesis and Test-Case Generation for Autonomous Systems
- 2023
-
Ingår i: Proceedings - 2023 IEEE 16th International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2023. - : Institute of Electrical and Electronics Engineers Inc.. - 9798350333350 ; , s. 18-27
-
Konferensbidrag (refereegranskat)abstract
- Autonomous systems are supposed to automatically plan their actions and execute the plan without human intervention. In this paper, we propose a model-based two-layer frame-work for policy synthesis and test-case generation for autonomous systems. At the high-level layer of the framework, we have two kinds of methods for synthesising policies whose correctness is guaranteed by model checking. The autonomous system's controller executes synthesised policies at the low-level layer. As the kinematics of autonomous systems is often nonlinear and the environment may influence the results of their actions, formally verifying the controllers is extremely difficult. We propose a novel method for generating test cases for the controllers at the low-level layer. The method employs reinforcement learning for test-case generation and model checking to ensure that the test cases faithfully realise the execution of the policy. The framework is designed in Uppaal Stratego, which integrates model checkers and algorithms for policy synthesis. Therefore, the framework separates concerns and seamlessly interchanges the information between two layers.
|
|
| 5. |
- Landin, Cristina, 1984-, et al.
(författare)
-
An Intelligent Monitoring Algorithm to Detect Dependencies between Test Cases in the Manual Integration Process
- 2023
-
Ingår i: 2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). - : IEEE. - 9798350333350 - 9798350333367 ; , s. 353-360
-
Konferensbidrag (refereegranskat)abstract
- Finding a balance between meeting test coverage and minimizing the testing resources is always a challenging task both in software (SW) and hardware (HW) testing. Therefore, employing machine learning (ML) techniques for test optimization purposes has received a great deal of attention. However, utilizing machine learning techniques frequently requires large volumes of valuable data to be trained. Although, the data gathering is hard and also expensive, manual data analysis takes most of the time in order to locate the source of failure once they have been produced in the so-called fault localization. Moreover, by applying ML techniques to historical production test data, relevant and irrelevant features can be found using strength association, such as correlation- and mutual information-based methods. In this paper, we use production data records of 100 units of a 5G radio product containing more than 7000 test results. The obtained results show that insightful information can be found after clustering the test results by their strength association, most linear and monotonic, which would otherwise be challenging to identify by traditional manual data analysis methods.
|
|
| 6. |
- Larsson, Jonatan, et al.
(författare)
-
Test Generation and Mutation Analysis of Energy Consumption using UPPAAL SMC and MATS
- 2023
-
Ingår i: 2023 IEEE INTERNATIONAL CONFERENCE ON SOFTWARE TESTING, VERIFICATION AND VALIDATION WORKSHOPS, ICSTW. - : IEEE COMPUTER SOC. - 9798350333350 ; , s. 186-189
-
Konferensbidrag (refereegranskat)abstract
- Testing is an essential process for ensuring the quality of the software. Designing software with as few errors as possible in most embedded systems is often critical. Resource usage is a significant concern for proper behaviour because of the very nature of embedded systems. To design energy-efficient systems, approaches are needed to catch desirable consumption points and correct them before deployment. Model-based testing can reduce testing effort, one testing method that allows for automatic test generation. However, this technique has yet to be studied extensively for revealing resource usage anomalies in embedded systems development. UPPAAL SMC is a statistical model-checking tool that can model the system's resource usage. This paper shows the tooling needed to achieve this and experimental results on automated test generation and selection using mutation analysis in UPPAAL SMC and how this is applied to a Brake by Wire industrial system. The evaluation shows that this approach, which we call MATS, is applicable and efficient for energy-based test generation.
|
|
| 7. |
- Ma, Yunpeng, et al.
(författare)
-
Automated and Systematic Digital Twins Testing for Industrial Processes
- 2023
-
Ingår i: Proceedings - 2023 IEEE 16th International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2023. - : Institute of Electrical and Electronics Engineers Inc.. - 9798350333350 ; , s. 149-158
-
Konferensbidrag (refereegranskat)abstract
- Digital twins (DT) of industrial processes have become increasingly important. They aim to digitally represent the physical world to help evaluate, optimize, and predict physical processes and behaviors. Therefore, DT is a vital tool to improve production automation through digitalization and becomes more sophisticated due to rapidly evolving simulation and modeling capabilities, integration of IoT sensors with DT, and high-capacity cloud/edge computing infrastructure. However, the fidelity and reliability of DT software are essential to represent the physical world. This paper shows an automated and systematic test architecture for DT that correlates DT states with real-time sensor data from a production line in the forging industry. Our evaluation shows that the architecture can significantly accelerate the automatic DT testing process and improve its reliability. A systematic online DT testing method can significantly detect the performance shift and continuously improve the DT's fidelity. The snapshot creation methodology and testing agent architecture can be an inspiration and can be generally applicable to other industrial processes that use DT to generalize their automated testing.
|
|
| 8. |
- Orgard, Jonathan, et al.
(författare)
-
Mutation Testing in Continuous Integration: An Exploratory Industrial Case Study
- 2023
-
Ingår i: Proceedings - 2023 IEEE 16th International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2023. - 9798350333350 ; , s. 324-333
-
Konferensbidrag (refereegranskat)abstract
- Despite its potential quality benefits, the cost of mutation testing and the immaturity of mutation tools for many languages have led to a lack of adoption in industrial software development. In an exploratory case study at Zenseact - a company in the automotive domain - we have explored how mutation testing could be effectively applied in a typical Continuous Integration-based workflow. We evaluated the capabilities of C++ mutation tools, and demonstrate their use in GitHub Actionsbased CI workflows. Our investigation reveals that Dextool and Mull could be used in a CI workflow. Additionally, we conducted an interview study to understand how developers would use mutation testing in their CI workflows. Based on our qualitative analysis and practices proposed in literature, we discuss recommendations to integrate mutation testing in a CI workflow. For instance, visualising trends in the mutation score enable practitioners to understand how test quality is evolving. Moreover, tools should have a balance between offering fast feedback and keeping or flagging relevant mutants. Lastly, practitioners raised the need that the mutation should be applied at commit level, and that developers inexperienced with mutation testing should be trained in the implications of the practice.
|
|
| 9. |
- Salari, Mikael Ebrahimi, et al.
(författare)
-
An Experiment in Requirements Engineering and Testing using EARS Notation for PLC Systems
- 2023
-
Ingår i: Proceedings - 2023 IEEE 16th International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2023. - : Institute of Electrical and Electronics Engineers Inc.. - 9798350333350 ; , s. 10-17
-
Konferensbidrag (refereegranskat)abstract
- Regulatory standards for engineering safety-critical systems often demand both traceable requirements and specification-based testing, during development. Requirements are often written in natural language, yet for specification purposes, this may be supplemented by formal or semi-formal descriptions, to increase clarity. However, the choice of notation of the latter is often constrained by the training, skills, and preferences of the designers.The Easy Approach to Requirements Syntax (EARS) addresses the inherent imprecision of natural language requirements with respect to potential ambiguity and lack of accuracy. This paper investigates requirement formalization using EARS and specification-based testing of embedded software written in the IEC 61131-3 language, a programming standard used for developing Programmable Logic Controllers (PLC). Further, we investigate, by means of an experiment, how human participants translate natural language requirements into EARS and how they use the latter to test PLC software. We report our observations during the experiments, including the type of EARS patterns participants use to structure natural language requirements and challenges during the specification phase, as well as present the results of testing based on EARS-formalized requirements.
|
|
