| 1. |
- Enoiu, Eduard Paul, et al.
(författare)
-
A Comparative Study of Manual and Automated Testing for Industrial Control Software
- 2017
-
Ingår i: Proceedings - 10th IEEE International Conference on Software Testing, Verification and Validation, ICST 2017. - 9781509060313 ; , s. 412-417
-
Konferensbidrag (refereegranskat)abstract
- Automated test generation has been suggested as a way of creating tests at a lower cost. Nonetheless, it is not very well studied how such tests compare to manually written ones in terms of cost and effectiveness. This is particularly true for industrial control software, where strict requirements on both specification-based testing and code coverage typically are met with rigorous manual testing. To address this issue, we conducted a case study in which we compared manually and automatically created tests. We used recently developed real-world industrial programs written in the IEC 61131-3, a popular programming language for developing industrial control systems using programmable logic controllers. The results show that automatically generated tests achieve similar code coverage as manually created tests, but in a fraction of the time (an average improvement of roughly 90%). We also found that the use of an automated test generation tool does not result in better fault detection in terms of mutation score compared to manual testing. Specifically, manual tests more effectively detect logical, timer and negation type of faults, compared to automatically generated tests. The results underscore the need to further study how manual testing is performed in industrial practice and the extent to which automated test generation can be used in the development of reliable systems.
|
|
| 2. |
- Enoiu, Eduard Paul, et al.
(författare)
-
A Comparative Study of Manual and Automated Testing in Industrial Embedded Software
-
Annan publikation (populärvet., debatt m.m.)abstract
- Testing is an important activity in engineering of industrial embedded software. In certain application domains (e.g., railway industry) engineering software is certified according to safety standards that require extensive software testing procedures to be applied for the development of reliable systems. Mutation analysis is a technique for creating faulty versions of a software for the purpose of examining the fault detection ability of a test suite. Mutation analysis has been used for evaluating existing test suites, but also for generating test suites that detect injected faults (i.e., mutation testing). To support developers in software testing, we propose a technique for producing test cases using an automated test generation approach that operates using mutation testing for software written in IEC 61131-3 language, a programming standard for safety-critical embedded software, commonly used for Programmable Logic Controllers (PLCs). This approach uses the Uppaal model checker and is based on a combined model that contains all the mutants and the original program. We applied this approach in a tool for testing industrial PLC programs and evaluated it in terms of cost and fault detection. For realistic validation we collected industrial experimental evidence on how mutation testing compares with manual testing as well as automated decision-coverage adequate test generation. In the evaluation, we used manually seeded faults provided by four industrial engineers. The results show that even if mutation-based test generation achieves better fault detection than automated decision coverage-based test generation, these mutation-adequate test suites are not better at detecting faults than manual test suites. However, the mutation-based test suites are significantly less costly to create, in terms of testing time, than manually created test suites. Our results suggest that the fault detection scores could be improved by considering some new and improved mutation operators (e.g., Feedback Loop Insertion Operator (FIO)) for PLC programs as well as higher-order mutations.
|
|
| 3. |
- Enoiu, Eduard Paul, et al.
(författare)
-
A Controlled Experiment in Testing of Safety-Critical Embedded Software
- 2016
-
Ingår i: Proceedings - 2016 IEEE International Conference on Software Testing, Verification and Validation, ICST 2016. - 9781509018260 ; , s. 1-11
-
Konferensbidrag (refereegranskat)abstract
- In engineering of safety critical systems, regulatory standards often put requirements on both traceable specification-based testing, and structural coverage on program units. Automated test generation techniques can be used to generate inputs to cover the structural aspects of a program. However, there is no conclusive evidence on how automated test generation compares to manual test design, or how testing based on the program implementation relates to specification-based testing. In this paper, we investigate specification-and implementation-based testing of embedded software written in the IEC 61131-3 language, a programming standard used in many embedded safety critical software systems. Further, we measure the efficiency and effectiveness in terms of fault detection. For this purpose, a controlled experiment was conducted, comparing tests created by a total of twenty-three software engineering master students. The participants worked individually on manually designing and automatically generating tests for two IEC 61131-3 programs. Tests created by the participants in the experiment were collected and analyzed in terms of mutation score, decision coverage, number of tests, and testing duration. We found that, when compared to implementation-based testing, specification-based testing yields significantly more effective tests in terms of the number of faults detected. Specifically, specification-based tests more effectively detect comparison and value replacement type of faults, compared to implementation-based tests. On the other hand, implementation-based automated test generation leads to fewer tests (up to 85% improvement) created in shorter time than the ones manually created based on the specification.
|
|
| 4. |
- Enoiu, Eduard Paul, et al.
(författare)
-
Automated Test Generation using Model-Checking: An Industrial Evaluation
- 2016
-
Ingår i: International Journal on Software Tools for Technology Transfer. - Germany : Springer. - 1433-2779 .- 1433-2787. ; 18:3, s. 335-353
-
Tidskriftsartikel (refereegranskat)abstract
- In software development, testers often focus on functional testing to validate implemented programs against their specifications. In safety critical software development, testers are also required to show that tests exercise, or cover, the structure and logic of the implementation. To achieve different types of logic coverage, various program artifacts such as decisions and conditions are required to be exercised during testing. Use of model-checking for structural test generation has been proposed by several researchers. The limited application to models used in practice and the state-space explosion can, however, impact model-checking and hence the process of deriving tests for logic coverage. Thus, there is a need to validate these approaches against relevant industrial systems such that more knowledge is built on how to efficiently use them in practice. In this paper, we present a tool-supported approach to handle software written in the Function Block Diagram language such that logic coverage criteria can be formalized and used by a model-checker to automatically generate tests. To this end, we conducted a study based on industrial use-case scenarios from Bombardier Transportation AB, showing how our toolbox COMPLETETEST can be applied to generate tests in software systems used in the safety-critical domain. To evaluate the approach, we applied the toolbox to 157 programs and found that it is efficient in terms of time required to generate tests that satisfy logic coverage and scales well for most of the programs.
|
|
| 5. |
- Enoiu, Eduard Paul, et al.
(författare)
-
Model-based test suite generation for function block diagrams using the UPPAAL model checker
- 2013
-
Ingår i: Proceedings - IEEE 6th International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2013. ; , s. 158-167
-
Konferensbidrag (refereegranskat)abstract
- A method for model-based test generation of safety-critical embedded applications using Programmable Logic Controllers and implemented in a programming language such as Function Block Diagram (FBD) is described. The FBD component model is based on the IEC 1131 standard and it is used primarily for embedded systems, in which timeliness is an important property to be tested. Our method involves the transformation of FBD programs with timed annotations into timed automata models which are used to automatically generate test suites. Specifically we demonstrate how to use model transformation for formalization and model checking of FBD programs using the UPPAAL tool. Many benefits emerge from this method, including the ability to automatically generate test suites from a formal model in order to ensure compliance to strict quality requirements including unit testing and specific coverage measurements. The approach is experimentally assessed on a train control system in terms of consumed resources.
|
|
| 6. |
|
|
| 7. |
- Enoiu, Eduard Paul, et al.
(författare)
-
MOS : An Integrated Model-based and Search-based Testing Tool for Function Block Diagrams
- 2013
-
Ingår i: 2013 1st International Workshop on Combining Modelling and Search-Based Software Engineering, CMSBSE 2013 - Proceedings. - 9781467362849 ; , s. 55-60
-
Konferensbidrag (refereegranskat)abstract
- In this paper we present a new testing tool for safety critical applications described in Function Block Diagram (FBD) language aimed to support both a model and a search-based approach. Many benefits emerge from this tool, including the ability to automatically generate test suites from an FBD program in order to comply to quality requirements such as component testing and specific coverage measurements. Search-based testing methods are used to generate test data based on executable code rather than the FBD program, alleviating any problems that may arise from the ambiguities that occur while creating FBD programs. Test cases generated by both approaches are executed and used as a way of cross validation. In the current work, we describe the architecture of the tool, its workflow process, and a case study in which the tool has been applied in a real industrial setting to test a train control management system.
|
|
| 8. |
- Enoiu, Eduard Paul, et al.
(författare)
-
Mutation-based test generation for PLC embedded software using model checking
- 2016
-
Ingår i: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). - Cham : Springer. - 9783319474427 ; , s. 155-171
-
Konferensbidrag (refereegranskat)abstract
- Testing is an important activity in engineering of industrial embedded software. In certain application domains (e.g., railway industry) engineering software is certified according to safety standards that require extensive software testing procedures to be applied for the development of reliable systems. Mutation analysis is a technique for creating faulty versions of a software for the purpose of examining the fault detection ability of a test suite. Mutation analysis has been used for evaluating existing test suites, but also for generating test suites that detect injected faults (i.e., mutation testing). To support developers in software testing, we propose a technique for producing test cases using an automated test generation approach that operates using mutation testing for software written in IEC 61131-3 language, a programming standard for safety-critical embedded software, commonly used for Programmable Logic Controllers (PLCs). This approach uses the Uppaal model checker and is based on a combined model that contains all the mutants and the original program. We applied this approach in a tool for testing industrial PLC programs and evaluated it in terms of cost and fault detection. For realistic validation we collected industrial experimental evidence on how mutation testing compares with manual testing as well as automated decision-coverage adequate test generation. In the evaluation, we used manually seeded faults provided by four industrial engineers. The results show that even if mutation-based test generation achieves better fault detection than automated decision coverage-based test generation, these mutation-adequate test suites are not better at detecting faults than manual test suites. However, the mutation-based test suites are significantly less costly to create, in terms of testing time, than manually created test suites. Our results suggest that the fault detection scores could be improved by considering some new and improved mutation operators (e.g., Feedback Loop Insertion Operator (FIO)) for PLC programs as well as higher-order mutations.
|
|
| 9. |
- Enoiu, Eduard Paul, et al.
(författare)
-
Towards the Analysis and Verification of EAST-ADL Models using UPPAAL PORT
- 2012
-
Rapport (övrigt vetenskapligt/konstnärligt)abstract
- A system’s architecture influence on the functions and other properties of embedded systems makes its high level analysis and verification very desirable. EAST-ADL is an architecture description language dedicated to automotive embedded system design with focus on structural and functional modeling. The behavioral description is not integrated within the execution semantics, which makes it harder to transform, analyze, and verify EAST-ADL models. Model-based techniques help address this issue by enabling automated transformation between different design models, and providing means for simulation and verification. We present a verification tool, called ViTAL, which provides the possibility to express the functional EAST-ADL behavior as timed automata models, which have precise semantics and can be formally verified. The ViTAL tool enables the transformation of EAST-ADL functional models to the UPPAAL PORT tool for model checking. This method improves the verification of functional and timing requirements in EAST-ADL, and makes it possible to identify dependencies and potential conflicts between different vehicle functions before the actual AUTOSAR implementation.
|
|
| 10. |
- Enoiu, Eduard Paul, et al.
(författare)
-
Using Logic Coverage to Improve Testing Function Block Diagrams
- 2013
-
Ingår i: Testing Software and Systems. - Berlin, Heidelberg : Springer Berlin Heidelberg. - 9783642417061 ; , s. 1-16
-
Bokkapitel (refereegranskat)abstract
- In model-driven development, testers are often focusing on functional model-level testing, enabling verification of design models against their specifications. In addition, in safety-critical software development, testers are required to show that tests cover the structure of the implementation. Testing cost and time savings could be achieved if the process of deriving test cases for logic coverage is automated and provided test cases are ready to be executed. The logic coverage artifacts, i.e., predicates and clauses, are required for different logic coverage, e.g., MC/DC. One way of dealing with test case generation for ensuring logic coverage is to approach it as a model-checking problem, such that model-checking tools automatically create test cases. We show how logic coverage criteria can be formalized and used by a model-checker to provide test cases for ensuring this coverage on safety-critical software described in the Function Block Diagram programming language. Based on our experiments, this approach, supported by a tool chain, is an applicable and useful way of generating test cases for covering Function Block Diagrams.
|
|
| 11. |
|
|
| 12. |
- Enoiu, Eduard Paul, et al.
(författare)
-
ViTAL : A Verification Tool for EAST-ADL Models using UPPAAL PORT
- 2012
-
Ingår i: Proceedings of the 17th IEEE International Conference on Engineering of Complex Computer Systems. - Paris, France. - 9782954181004 ; , s. 328-337
-
Konferensbidrag (refereegranskat)abstract
- A system’s architecture influence on the functions and other properties of embedded systems makes its high level analysis and verification very desirable. EAST-ADL is an architecture description language dedicated to automotive embedded system design with focus on structural and functional modeling. The behavioral description is not integrated within the execution semantics, which makes it harder to transform, analyze, and verify EAST-ADL models. Model-based techniques help address this issue by enabling automated transformation between different design models, and providing means for simulation and verification. We present a verification tool, called ViTAL, which provides the possibility to express the functional EAST-ADL behavior as timed automata models, which have precise semantics and can be formally verified. The ViTAL tool enables the transformation of EAST-ADL functional models to the UPPAAL PORT tool for model checking. This method improves the verification of functional and timing requirements in EAST-ADL, and makes it possible to identify dependencies and potential conflicts between different vehicle functions before the actual AUTOSAR implementation.
|
|
| 13. |
- Gu, Rong, et al.
(författare)
-
Synthesis and Verification of Mission Plans for Multiple Autonomous Agents under Complex Road Conditions
-
Annan publikation (övrigt vetenskapligt/konstnärligt)abstract
- Mission planning for multi-agent autonomous systems aims to generate feasible and optimal mission plans that satisfy the given requirements. In this article, we propose a mission-planning methodology that combines (i) a path-planning algorithm for synthesizing path plans that are safe in environments with complex road conditions, and (ii) a task-scheduling method for synthesizing task plans that schedule the tasks in the right and fastest order, taking into account the planned paths. The task-scheduling method is based on model checking, which provides means of automatically generating task execution orders that satisfy the requirements and ensure the correctness and efficiency of the plans by construction. We implement our approach in a tool named MALTA, which offers a user-friendly GUI for configuring mission requirements, a module for path planning, an integration with the model checker UPPAAL, and functions for automatic generation of formal models, and parsing of the execution traces of models. Experiments with the tool demonstrate its applicability and performance in various configurations of an industrial case study of an autonomous quarry. We also show the adaptability of our tool by employing it on a special case of the industrial case study.
|
|
| 14. |
- Kang, Eun-Young, et al.
(författare)
-
A Methodology for Formal Analysis and Verification of EAST-ADL Models
- 2013
-
Ingår i: Reliability Engineering & System Safety. - : Elsevier. - 0951-8320 .- 1879-0836. ; 120:Special Issue, s. 127-138
-
Tidskriftsartikel (refereegranskat)abstract
- The architectural design of embedded software has a direct impact on the final implementation, with respect to performance and other quality attributes. Therefore, guaranteeing that an architectural model meets the specified requirements is beneficial for detecting software flaws early in the development process. In this paper, we present a formal modeling and verification methodology for safety-critical automotive products that are originally described in the domain-specific architectural language East-adl. We propose a model-based approach that integrates the architectural models with component-aware model checking, and describe its tool support called ViTAL. The functional and timing behavior of each function block in the East-adl model, as well as the interactions between function blocks are formally captured and expressed as Timed Automata models, which have precise semantics and can be formally verified with ViTAL. Furthermore, we show how our approach, supported by ViTAL, can be used to formally prove that the East-adl system model fulfills the specified real-time requirements and behavioral constraints. We demonstrate that the approach improves the modeling and verification capability of East-adl and identifies dependencies, as well as potential conflicts between different automotive functions before implementation. The method is substantiated by verifying an automotive braking system model, with respect to particular functional and timing requirements.
|
|
| 15. |
- Abbas, Muhammad, et al.
(författare)
-
Automated Reuse Recommendation of Product Line Assets Based on Natural Language Requirements
- 2020
-
Ingår i: Lecture Notes in Computer Science. - Cham : Springer Science and Business Media Deutschland GmbH. - 9783030646936 ; , s. 173-189, s. 173-189, s. 173-189
-
Konferensbidrag (refereegranskat)abstract
- Software product lines (SPLs) are based on reuse rationale to aid quick and quality delivery of complex products at scale. Deriving a new product from a product line requires reuse analysis to avoid redundancy and support a high degree of assets reuse. In this paper, we propose and evaluate automated support for recommending SPL assets that can be reused to realize new customer requirements. Using the existing customer requirements as input, the approach applies natural language processing and clustering to generate reuse recommendations for unseen customer requirements in new projects. The approach is evaluated both quantitatively and qualitatively in the railway industry. Results show that our approach can recommend reuse with 74% accuracy and 57.4% exact match. The evaluation further indicates that the recommendations are relevant to engineers and can support the product derivation and feasibility analysis phase of the projects. The results encourage further study on automated reuse analysis on other levels of abstractions.
|
|
| 16. |
|
|
| 17. |
- Abbas, Muhammad, et al.
(författare)
-
Is Requirements Similarity a Good Proxy for Software Similarity? : An Empirical Investigation in Industry
- 2021
-
Ingår i: <em>Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) </em>27th International Working Conference on Requirements Engineering: Foundation for Software Quality, REFSQ 2021, 12 April 2021 - 15 April 2021. - Cham : Springer Science and Business Media Deutschland GmbH. - 9783030731274 ; , s. 3-18, s. 3-18
-
Konferensbidrag (refereegranskat)abstract
- [Context and Motivation] Content-based recommender systems for requirements are typically built on the assumption that similar requirements can be used as proxies to retrieve similar software. When a new requirement is proposed by a stakeholder, natural language processing (NLP)-based similarity metrics can be exploited to retrieve existing requirements, and in turn identify previously developed code. [Question/problem] Several NLP approaches for similarity computation are available, and there is little empirical evidence on the adoption of an effective technique in recommender systems specifically oriented to requirements-based code reuse. [Principal ideas/results] This study compares different state-of-the-art NLP approaches and correlates the similarity among requirements with the similarity of their source code. The evaluation is conducted on real-world requirements from two industrial projects in the railway domain. Results show that requirements similarity computed with the traditional tf-idf approach has the highest correlation with the actual software similarity in the considered context. Furthermore, results indicate a moderate positive correlation with Spearman’s rank correlation coefficient of more than 0.5. [Contribution] Our work is among the first ones to explore the relationship between requirements similarity and software similarity. In addition, we also identify a suitable approach for computing requirements similarity that reflects software similarity well in an industrial context. This can be useful not only in recommender systems but also in other requirements engineering tasks in which similarity computation is relevant, such as tracing and categorization.
|
|
| 18. |
- Abbas, Muhammad, et al.
(författare)
-
Keywords-based test categorization for Extra-Functional Properties
- 2020
-
Ingår i: 2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). - : IEEE. - 9781728110752 ; , s. 153-156
-
Konferensbidrag (refereegranskat)abstract
- Categorizing existing test specifications can provide insights on coverage of the test suite to extra-functional properties. Manual approaches for test categorization can be time-consuming and prone to error. In this short paper, we propose a semi-automated approach for semantic keywords-based textual test categorization for extra-functional properties. The approach is the first step towards coverage-based test case selection based on extra-functional properties. We report a preliminary evaluation of industrial data for test categorization for safety aspects. Results show that keyword-based approaches can be used to categorize tests for extra-functional properties and can be improved by considering contextual information of keywords.
|
|
| 19. |
- Abbas, Muhammad, et al.
(författare)
-
MBRP : Model-Based Requirements Prioritization Using PageRank Algorithm
- 2019
-
Ingår i: 2019 26th Asia-Pacific Software Engineering Conference (APSEC). - : IEEE conference proceedings. - 9781728146485 ; , s. 31-38
-
Konferensbidrag (refereegranskat)abstract
- Requirements prioritization plays an important role in driving project success during software development. Literature reveals that existing requirements prioritization approaches ignore vital factors such as interdependency between requirements. Existing requirements prioritization approaches are also generally time-consuming and involve substantial manual effort. Besides, these approaches show substantial limitations in terms of the number of requirements under consideration. There is some evidence suggesting that models could have a useful role in the analysis of requirements interdependency and their visualization, contributing towards the improvement of the overall requirements prioritization process. However, to date, just a handful of studies are focused on model-based strategies for requirements prioritization, considering only conflict-free functional requirements. This paper uses a meta-model-based approach to help the requirements analyst to model the requirements, stakeholders, and inter-dependencies between requirements. The model instance is then processed by our modified PageRank algorithm to prioritize the given requirements. An experiment was conducted, comparing our modified PageRank algorithm's efficiency and accuracy with five existing requirements prioritization methods. Besides, we also compared our results with a baseline prioritized list of 104 requirements prepared by 28 graduate students. Our results show that our modified PageRank algorithm was able to prioritize the requirements more effectively and efficiently than the other prioritization methods.
|
|
| 20. |
- Abbas, Muhammad, et al.
(författare)
-
On the relationship between similar requirements and similar software : A case study in the railway domain
- 2023
-
Ingår i: Requirements Engineering. - : Springer Science and Business Media Deutschland GmbH. - 0947-3602 .- 1432-010X. ; 28, s. 23-47
-
Tidskriftsartikel (refereegranskat)abstract
- Recommender systems for requirements are typically built on the assumption that similar requirements can be used as proxies to retrieve similar software. When a stakeholder proposes a new requirement, natural language processing (NLP)-based similarity metrics can be exploited to retrieve existing requirements, and in turn, identify previously developed code. Several NLP approaches for similarity computation between requirements are available. However, there is little empirical evidence on their effectiveness for code retrieval. This study compares different NLP approaches, from lexical ones to semantic, deep-learning techniques, and correlates the similarity among requirements with the similarity of their associated software. The evaluation is conducted on real-world requirements from two industrial projects from a railway company. Specifically, the most similar pairs of requirements across two industrial projects are automatically identified using six language models. Then, the trace links between requirements and software are used to identify the software pairs associated with each requirements pair. The software similarity between pairs is then automatically computed with JPLag. Finally, the correlation between requirements similarity and software similarity is evaluated to see which language model shows the highest correlation and is thus more appropriate for code retrieval. In addition, we perform a focus group with members of the company to collect qualitative data. Results show a moderately positive correlation between requirements similarity and software similarity, with the pre-trained deep learning-based BERT language model with preprocessing outperforming the other models. Practitioners confirm that requirements similarity is generally regarded as a proxy for software similarity. However, they also highlight that additional aspect comes into play when deciding software reuse, e.g., domain/project knowledge, information coming from test cases, and trace links. Our work is among the first ones to explore the relationship between requirements and software similarity from a quantitative and qualitative standpoint. This can be useful not only in recommender systems but also in other requirements engineering tasks in which similarity computation is relevant, such as tracing and change impact analysis.
|
|
| 21. |
- Abbas, Muhammad, et al.
(författare)
-
Product line adoption in industry : an experience report from the railway domain
- 2020
-
Ingår i: SPLC '20: Proceedings of the 24th ACM Conference on Systems and Software Product Line: Volume A. - New York, NY, USA : Association for Computing Machinery. - 9781450375696 ; , s. 130-141, s. 130-141
-
Konferensbidrag (refereegranskat)abstract
- The software system controlling a train is typically deployed on various hardware architectures and must process various signals across those deployments. The increase of such customization scenarios and the needed adherence of the software to various safety standards in different application domains has led to the adoption of product line engineering within the railway domain. This paper explores the current state-of-practice of software product line development within a team developing industrial embedded software for a train propulsion control system. Evidence is collected using a focus group session with several engineers and through inspection of archival data. We report several benefits and challenges experienced during product line adoption and deployment. Furthermore, we identify and discuss improvement opportunities, focusing mainly on product line evolution and test automation.
|
|
| 22. |
- Abbas, Muhammad, et al.
(författare)
-
Requirements-Driven Reuse Recommendation
- 2021
-
Ingår i: Proceedings of the 25th ACM International Systems and Software Product Line Conference - Volume A. - New York, NY, USA : Association for Computing Machinery.
-
Konferensbidrag (refereegranskat)abstract
- This tutorial explores requirements-based reuse recommendation for product line assets in the context of clone-and-own product lines.
|
|
| 23. |
- Abbas, Muhammad
(författare)
-
Requirements-Level Reuse Recommendation and Prioritization of Product Line Assets
- 2021
-
Licentiatavhandling (övrigt vetenskapligt/konstnärligt)abstract
- Software systems often target a variety of different market segments. Targeting varying customer requirements requires a product-focused development process. Software Product Line (SPL) engineering is one possible approach based on reuse rationale to aid quick delivery of quality product variants at scale. SPLs reuse common features across derived products while still providing varying configuration options. The common features, in most cases, are realized by reusable assets. In practice, the assets are reused in a clone-and-own manner to reduce the upfront cost of systematic reuse. Besides, the assets are implemented in increments, and requirements prioritization also has to be done. In this context, the manual reuse analysis and prioritization process become impractical when the number of derived products grows. Besides, the manual reuse analysis process is time-consuming and heavily dependent on the experience of engineers.In this licentiate thesis, we study requirements-level reuse recommendation and prioritization for SPL assets in industrial settings. We first identify challenges and opportunities in SPLs where reuse is done in a clone-and-own manner. We then focus on one of the identified challenges: requirements-based SPL assets reuse and provide automated support for identifying reuse opportunities for SPL assets based on requirements. Finally, we provide automated support for requirements prioritization in the presence of dependencies resulting from reuse.
|
|
| 24. |
|
|
| 25. |
- Abbaspour Asadollah, Sara, et al.
(författare)
-
A Study on Concurrency Bugs in an Open Source Software
- 2016
-
Ingår i: IFIP Advances in Information and Communication Technology, vol. 472. - Cham : Springer International Publishing. - 9783319392240 ; , s. 16-31
-
Konferensbidrag (refereegranskat)abstract
- Concurrent programming puts demands on software debugging and testing, as concurrent software may exhibit problems not present in sequential software, e.g., deadlocks and race conditions. In aiming to increase efficiency and effectiveness of debugging and bug-fixing for concurrent software, a deep understanding of concurrency bugs, their frequency and fixingtimes would be helpful. Similarly, to design effective tools and techniques for testing and debugging concurrent software understanding the differences between non-concurrency and concurrency bugs in real-word software would be useful.
|
|
| 26. |
- Ahmad, Azeem, et al.
(författare)
-
An Industrial Study on the Challenges and Effects of Diversity-Based Testing in Continuous Integration
- 2023
-
Ingår i: IEEE International Conference on Software Quality, Reliability and Security, QRS. - 2693-9177. - 9798350319583
-
Konferensbidrag (refereegranskat)abstract
- Many test prioritisation techniques have been proposed in order to improve test effectiveness of Continuous Integration (CI) pipelines. Particularly, diversity-based testing (DBT) has shown promising and competitive results to improve test effectiveness. However, the technical and practical challenges of introducing test prioritisation in CI pipelines are rarely discussed, thus hindering the applicability and adoption of those proposed techniques. This research builds on our prior work in which we evaluated diversity-based techniques in an industrial setting. This work investigates the factors that influence the adoption of DBT both in connection to improvements in test cost-effectiveness, as well as the process and human related challenges to transfer and use DBT prioritisation in CI pipelines. We report on a case study considering the CI pipeline of Axis Communications in Sweden. We performed a thematic analysis of a focus group interview with senior practitioners at the company to identify the challenges and perceived benefits of using test prioritisation in their test process. Our thematic analysis reveals a list of ten challenges and seven perceived effects of introducing test prioritisation in CI cycles. For instance, our participants emphasized the importance of introducing comprehensible and transparent techniques that instill trust in its users. Moreover, practitioners prefer techniques compatible with their current test infrastructure (e.g., test framework and environments) in order to reduce instrumentation efforts and avoid disrupting their current setup. In conclusion, we have identified tradeoffs between different test prioritisation techniques pertaining to the technical, process and human aspects of regression testing in CI. We summarize those findings in a list of seven advantages that refer to specific stakeholder interests and describe the effects of adopting DBT in CI pipelines.
|
|
| 27. |
- Ahmad, A., et al.
(författare)
-
The Comparative Evaluation of Test Prioritization Approaches in an Industrial Study
- 2023
-
Ingår i: Proc. - IEEE Int. Conf. Softw. Qual., Reliab., Secur. Companion, QRS-C. - : Institute of Electrical and Electronics Engineers Inc.. - 9798350359398 ; , s. 35-44
-
Konferensbidrag (refereegranskat)abstract
- Many test prioritisation techniques have been proposed in order to improve test effectiveness of Continuous Integration (CI) pipelines. Particularly, diversity-based testing (DBT) has shown promising and competitive results to improve test effectiveness. We report on a case study considering the CI pipeline of Axis Communications in Sweden. We compared three different prioritisation approaches (i.e., diversity, failure history and time) in terms of their impact on coverage, failure detection rates and reduction on test execution time. Our results reveal that DBT is the best candidate to provide feature coverage, whereas failure rate prioritisation yields better failure coverage. Time-based prioritisation is not a reliable approach to provide cost-effective testing. Moreover, DBT would allow stakeholders to receive quick feedback on many combinations of integrated features to verify their code changes. Our participants report that developers are mainly interested in: (i) receiving quick feedback on a high combination of integrated features to verify their code changes, and (ii) associate their test suites to confidence scores representing the risk of missing failures given that fewer tests are executed.
|
|
| 28. |
- Ahmed, Bestoun S., 1982-, et al.
(författare)
-
An evaluation of Monte Carlo-based hyper-heuristic for interaction testing of industrial embedded software applications
- 2020
-
Ingår i: Soft Computing - A Fusion of Foundations, Methodologies and Applications. - : Springer. - 1432-7643 .- 1433-7479. ; 24:18, s. 13929-13954
-
Tidskriftsartikel (refereegranskat)abstract
- Hyper-heuristic is a new methodology for the adaptive hybridization of meta-heuristic algorithms to derive a general algorithm for solving optimization problems. This work focuses on the selection type of hyper-heuristic, called the exponential Monte Carlo with counter (EMCQ). Current implementations rely on the memory-less selection that can be counterproductive as the selected search operator may not (historically) be the best performing operator for the current search instance. Addressing this issue, we propose to integrate the memory into EMCQ for combinatorial t-wise test suite generation using reinforcement learning based on the Q-learning mechanism, called Q-EMCQ. The limited application of combinatorial test generation on industrial programs can impact the use of such techniques as Q-EMCQ. Thus, there is a need to evaluate this kind of approach against relevant industrial software, with a purpose to show the degree of interaction required to cover the code as well as finding faults. We applied Q-EMCQ on 37 real-world industrial programs written in Function Block Diagram (FBD) language, which is used for developing a train control management system at Bombardier Transportation Sweden AB. The results show that Q-EMCQ is an efficient technique for test case generation. Addition- ally, unlike the t-wise test suite generation, which deals with the minimization problem, we have also subjected Q-EMCQ to a maximization problem involving the general module clustering to demonstrate the effectiveness of our approach. The results show the Q-EMCQ is also capable of outperforming the original EMCQ as well as several recent meta/hyper-heuristic including modified choice function, Tabu high-level hyper-heuristic, teaching learning-based optimization, sine cosine algorithm, and symbiotic optimization search in clustering quality within comparable execution time.
|
|
| 29. |
- Aronsson Karlsson, Viktor, et al.
(författare)
-
Automation of the creation and execution of system level hardware-in-loop tests through model-based testing
- 2022
-
Ingår i: A-TEST - Proc. Int. Workshop Autom. Test Case Des., Select., Eval., co-located ESEC/FSE. - New York, NY, USA : Association for Computing Machinery, Inc. - 9781450394529 ; , s. 9-16
-
Konferensbidrag (refereegranskat)abstract
- In this paper, we apply model-based testing (MBT) to automate the creation of hardware-in-loop (HIL) test cases. In order to select MBT tools, different tools' properties were compared to each other through a literature study, with the result of selecting GraphWalker and MoMuT tools to be used in an industrial case study. The results show that the generated test cases perform similarly to their manual counterparts regarding how the test cases achieved full requirements coverage. When comparing the effort needed for applying the methods, a comparable effort is required for creating the first iteration, while with every subsequent update, MBT will require less effort compared to the manual process. Both methods achieve 100% requirements coverage, and since manual tests are created and executed by humans, some requirements are favoured over others due to company demands, while MBT tests are generated randomly. In addition, a comparison between the used tools showcased the differences in the models' design and their test case generation. The comparison showed that GraphWalker has a more straightforward design method and is better suited for smaller systems, while MoMuT can handle more complex systems but has a more involved design method.
|
|
| 30. |
- Ayerdi, J., et al.
(författare)
-
Towards a Taxonomy for Eliciting Design-Operation Continuum Requirements of Cyber-Physical Systems
- 2020
-
Ingår i: Proceedings of the IEEE International Conference on Requirements Engineering. - : IEEE Computer Society. - 9781728174389 ; , s. 280-290
-
Konferensbidrag (övrigt vetenskapligt/konstnärligt)abstract
- Software systems that are embedded in autonomous Cyber-Physical Systems (CPSs) usually have a large life-cycle, both during its development and in maintenance. This software evolves during its life-cycle in order to incorporate new requirements, bug fixes, and to deal with hardware obsolescence. The current process for developing and maintaining this software is very fragmented, which makes developing new software versions and deploying them in the CPSs extremely expensive. In other domains, such as web engineering, the phases of development and operation are tightly connected, making it possible to easily perform software updates of the system, and to obtain operational data that can be analyzed by engineers at development time. However, in spite of the rise of new communication technologies (e.g., 5G) providing an opportunity to acquire Design-Operation Continuum Engineering methods in the context of CPSs, there are still many complex issues that need to be addressed, such as the ones related with hardware-software co-design. Therefore, the process of Design-Operation Continuum Engineering for CPSs requires substantial changes with respect to the current fragmented software development process. In this paper, we build a taxonomy for Design-Operation Continuum Engineering of CPSs based on case studies from two different industrial domains involving CPSs (elevation and railway). This taxonomy is later used to elicit requirements from these two case studies in order to present a blueprint on adopting Design-Operation Continuum Engineering in any organization developing CPSs.
|
|
| 31. |
- Barrett, Ayodele, et al.
(författare)
-
On the Current State of Academic Software Testing Education in Sweden
- 2023
-
Ingår i: Proceedings - 2023 IEEE 16th International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2023. - : Institute of Electrical and Electronics Engineers Inc.. - 9798350333350 ; , s. 397-404
-
Konferensbidrag (refereegranskat)abstract
- Well-trained software development personnel, in the art and science of software testing, will effectively and efficiently develop quality software products with potentially fewer, less-critical defects. Thus software testing education is considered to be an important part of curricula for a university degree in Computer Science or Information Systems. The objective of this paper is to determine how much dedicated knowledge in the field of software testing is taught within Swedish universities. To achieve this objective, a systematic search of syllabi for software testing-related courses was done. From 25 Swedish universities offering Computer Science (or related) degrees, 14 currently offer dedicated courses in software testing. Some findings include: 32% of the individual courses were offered at the undergraduate level; 28% of the universities offer courses for specialised testing training; and, for the vast majority of the universities, dedicated software testing courses account for about 5% of the total degree credits. While some universities fare better than others, the overall state of academic software testing education in Sweden is limited but promising.
|
|
| 32. |
- Bashir, Sarmad, et al.
(författare)
-
Requirement or Not, That is the Question : A Case from the Railway Industry
- 2023
-
Ingår i: <em>Lecture Notes in Computer Science. </em>Volume 13975. Pages 105 - 121 2023. - : Springer Science and Business Media Deutschland GmbH. - 9783031297854 ; , s. 105-121
-
Konferensbidrag (refereegranskat)abstract
- Requirements in tender documents are often mixed with other supporting information. Identifying requirements in large tender documents could aid the bidding process and help estimate the risk associated with the project. Manual identification of requirements in large documents is a resource-intensive activity that is prone to human error and limits scalability. This study compares various state-of-the-art approaches for requirements identification in an industrial context. For generalizability, we also present an evaluation on a real-world public dataset. We formulate the requirement identification problem as a binary text classification problem. Various state-of-the-art classifiers based on traditional machine learning, deep learning, and few-shot learning are evaluated for requirements identification based on accuracy, precision, recall, and F1 score. Results from the evaluation show that the transformer-based BERT classifier performs the best, with an average F1 score of 0.82 and 0.87 on industrial and public datasets, respectively. Our results also confirm that few-shot classifiers can achieve comparable results with an average F1 score of 0.76 on significantly lower samples, i.e., only 20% of the data. There is little empirical evidence on the use of large language models and few-shots classifiers for requirements identification. This paper fills this gap by presenting an industrial empirical evaluation of the state-of-the-art approaches for requirements identification in large tender documents. We also provide a running tool and a replication package for further experimentation to support future research in this area. © 2023, The Author(s)
|
|
| 33. |
- Bergström, Henning, et al.
(författare)
-
Using Timed Base-Choice Coverage Criterion for Testing Industrial Control Software
- 2017
-
Ingår i: Proceedings - 10th IEEE International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2017. - : Institute of Electrical and Electronics Engineers Inc.. - 9781509066766 ; , s. 216-219
-
Konferensbidrag (refereegranskat)abstract
- The base-choice criterion was proposed as a suitable technique for testing software based on its nominal choice of input parameters. Test cases are created based on this strategy by varying the values of one input parameter at a time while keeping the values of the other parameters fixed on the base choice. However, this strategy might not be as effective when used on industrial control software for testing timed behavior. We propose to incorporate time as another parameter when generating and executing tests by defining the timed base-choice coverage criterion. We performed an empirical evaluation using 11 industrial programs written in the IEC 61131-3 programming language. We found that tests generated for timed base-choice criterion show better code coverage (7% improvement) and fault detection (27% improvement) in terms of mutation score than tests satisfying base-choice coverage criterion. The results demonstrate the feasibility of applying timed base-choice criterion for testing industrial control software.
|
|
| 34. |
- Charbachi, Peter, et al.
(författare)
-
Can Pairwise Testing Perform Comparably to Manually Handcrafted Testing Carried Out by Industrial Engineers?
- 2017
-
Ingår i: Proceedings - 2017 IEEE International Conference on Software Quality, Reliability and Security Companion, QRS-C 2017. - : Institute of Electrical and Electronics Engineers Inc.. - 9781538620724 ; , s. 92-99
-
Konferensbidrag (refereegranskat)abstract
- Testing is an important activity in engineering of industrial software. For such software, testing is usually performed manually by handcrafting test suites based on specific design techniques and domain-specific experience. To support developers in testing, different approaches for producing good test suites have been proposed. In the last couple of years combinatorial testing has been explored with the goal of automatically combining the input values of the software based on a certain strategy. Pairwise testing is a combinatorial technique used to generate test suites by varying the values of each pair of input parameters to a system until all possible combinations of those parameters are created. There is some evidence suggesting that these kinds of techniques are efficient and relatively good at detecting software faults. Unfortunately, there is little experimental evidence on the comparison of these combinatorial testing techniques with, what is perceived as, rigorous manually handcrafted testing. In this study we compare pairwise test suites with test suites created manually by engineers for 45 industrial programs. The test suites were evaluated in terms of fault detection, code coverage and number of tests. The results of this study show that pairwise testing, while useful for achieving high code coverage and fault detection for the majority of the programs, is almost as effective in terms of fault detection as manual testing. The results also suggest that pairwise testing is just as good as manual testing at fault detection for 64% of the programs.
|
|
| 35. |
- Cusmaru, Alexandru, et al.
(författare)
-
How aware are we of our biases in testing?
- 2022
-
Ingår i: 30th Eurostar software testing conference.
-
Konferensbidrag (övrigt vetenskapligt/konstnärligt)abstract
- Unconscious biases are hard-wired behaviors that influence testers and can set them on an incorrect course of action. If we understand how these biases affect testers' everyday behavior, we can attempt to mitigate them, then develop more effective tools and strategies to help testers avoid the pitfalls of cognitive biases. But to what extent are testers aware that cognitive biases influence their work? This was the primary question we sought to answer in our research. We developed a questionnaire survey, designed to reveal the extent of software testers’ knowledge about cognitive biases, plus their awareness of the influence biases have on testing. We reached out to software professionals working in different environments and gathered valid data from approximately 60 practitioners. Our results suggest that professionals are indeed aware of biases. Specifically, they are aware of preconceptions such as confirmation bias, fixation, and convenience. In addition, optimism, ownership, and blissful ignorance were other common biases. In common with other research, we observed that people tend to identify more cognitive biases in others than in their own judgments and actions, indicating a vulnerability to bias blind spot.
|
|
| 36. |
- de Oliveira Neto, Francisco Gomes, et al.
(författare)
-
Improving continuous integration with similarity-based test case selection
- 2018
-
Ingår i: Proceedings of the 13th International Workshop on Automation of Software Test. - New York : ACM Digital Library. - 0270-5257. - 9781450357432 ; , s. 39-45
-
Konferensbidrag (refereegranskat)abstract
- Automated testing is an essential component of Continuous Integration (CI) and Delivery (CD), such as scheduling automated test sessions on overnight builds. That allows stakeholders to execute entire test suites and achieve exhaustive test coverage, since running all tests is often infeasible during work hours, i.e., in parallel to development activities. On the other hand, developers also need test feedback from CI servers when pushing changes, even if not all test cases are executed. In this paper we evaluate similarity-based test case selection (SBTCS) on integration-level tests executed on continuous integration pipelines of two companies. We select test cases that maximise diversity of test coverage and reduce feedback time to developers. Our results confirm existing evidence that SBTCS is a strong candidate for test optimisation, by reducing feedback time (up to 92% faster in our case studies) while achieving full test coverage using only information from test artefacts themselves.
|
|
| 37. |
- Enoiu, Eduard Paul, et al.
(författare)
-
A Design Tool for Service-oriented Systems
- 2013
-
Ingår i: Electronic Notes in Theoretical Computer Science. - : Elsevier BV. - 1571-0661. ; 295, s. 95-100
-
Tidskriftsartikel (övrigt vetenskapligt/konstnärligt)abstract
- In this paper we present a modeling and analysis tool for service-oriented systems. The tool enables graphical modeling of service-based systems, within the resource-aware timed behavioral language Remes, as well as a textual system description. We have developed a graphical environment where services can be composed as desired by the user, together with a textual service composition interface in which compositions can also be checked for correctness. We also provide automated traceability between the two design interfaces, which results in a tool that enhances the potential of system design by intuitive service manipulation. The paper presents the design principles, infrastructure, and the user interface of our tool.
|
|
| 38. |
- Enoiu, Eduard Paul, et al.
(författare)
-
Enablers and impediments for collaborative research in software testing : An empirical exploration
- 2014
-
Ingår i: WISE 2014 - Proceedings of the 2014 ACM International Workshop on Long-Term Industrial Collaboration on Software Engineering, Co-located with ASE 2014. - New York, NY, USA : ACM. - 9781450330459 ; , s. 49-54
-
Konferensbidrag (refereegranskat)abstract
- When it comes to industrial organizations, current collaboration efforts in software engineering research are very often kept in-house, depriving these organizations off the skills necessary to build independent collaborative research. The current trend, towards empirical software engineering research, requires certain standards to be established which would guide these collaborative efforts in creating a strong partnership that promote independent, evidence-based, software engineering research. This paper examines key enabling factors for an effcient and effective industry-academia collaboration in the software testing domain. A major finding of the research was that while technology is a strong enabler to better collaboration, it must be complemented with industrial openness to disclose research results and the use of a dedicated tooling platform. We use as an example an automated test generation approach that has been developed in the last two years collaboratively with Bombardier Transportation AB in Sweden.
|
|
| 39. |
- Enoiu, Eduard Paul, PhD, et al.
(författare)
-
Model Testing of Complex Embedded Systems using EAST-ADL and Energy-Aware Mutations
- 2020
-
Ingår i: Designs. - Sweden : MDPI AG. - 2411-9660. ; 4:1, s. 1-18
-
Tidskriftsartikel (refereegranskat)abstract
- Nowadays, embedded systems are increasingly complex, meaning that traditional testing methods are costly to use and infeasible to directly apply due to the complex interactions between hardware and software. Modern embedded systems are also demanded to function based on low-energy computing. Hence, testing the energy usage is increasingly important. Artifacts produced during the development of embedded systems, such as architectural descriptions, are beneficial abstractions of the system’s complex structure and behavior. Electronic Architecture and Software Tools Architecture Description Language (EAST-ADL) is one such example of a domain-specific architectural language targeting the automotive industry. In this paper, we propose a method for testing design models using EAST-ADL architecture mutations. We show how fault-based testing can be used to generate, execute and select tests using energy-aware mutants-- syntactic changes in the architectural description, used to mimic naturally occurring energy faults. Our goal is to improve testing of complex embedded systems by moving the testing bulk from the actual systems to models of their behaviors and non-functional requirements. We combine statistical model-checking, increasingly used in quality assurance of embedded systems, with EAST-ADL architectural models and mutation testing to drive the search for faults. We show the results of applying this method on an industrial-sized system developed by Volvo GTT. The results indicate that model testing of EAST-ADL architectural models can reduce testing complexity by bringing early and cost-effective automation.
|
|
| 40. |
- Enoiu, Eduard Paul, PhD
(författare)
-
Teaching software testing to industrial practitioners using distance and web-based learning
- 2020
-
Ingår i: Lect. Notes Comput. Sci.. - Cham : Springer. - 9783030576622 ; , s. 73-87
-
Konferensbidrag (refereegranskat)abstract
- Software testing is a business-critical process used by private and public organizations and an important source of market competitiveness. Employees of these organizations are facing tough competition and are required to be able to maintain and develop their skills and knowledge in software testing. In the education market, many commercial courses and certifications are available for industrial engineers who wish to improve their skills in software development. Nevertheless, there is a lack of access to world-leading research within the software testing field in these commercial courses that supports the companies’ innovation in software testing. As an alternative, universities are approaching this challenge by developing academic courses on software testing that can suit professionals who need to be able to combine work and studies. This study highlights several good approaches and challenges in developing and teaching three distance web-based software testing courses targeting test practitioners. The proposed approaches for enhancing teaching of software testing in an online setting for industrial practitioners are: active participation at the student’s pace, inclusion of software testing artifacts from the student’s organization as part of assignments, continuous access to online materials, the use of short video materials on testing theory, and setting clear expectations for performing online test design assignments. Finally, several challenges have been identified: poor feedback on assignments, distances between students and teachers, the use of non-realistic assignments and the difficulty for industrial practitioners to complete academic assignments each week. Future work is needed to explore these results in practice, for example on how to shorten distances between students and teachers, as well as how to enhance the inclusion of real-world testing artifacts in course assignments. © Springer Nature Switzerland AG 2020.
|
|
| 41. |
- Enoiu, Eduard Paul, PhD, et al.
(författare)
-
Test Agents : Adaptive, Autonomous and Intelligent Test Cases
- 2018
-
Annan publikation (övrigt vetenskapligt/konstnärligt)abstract
- Growth of software size, lack of resources to perform regression testing, and failure to detect bugs faster have seen increased reliance on continuous integration and test automation. Even with greater hardware and software resources dedicated to test automation, software testing is faced with enormous challenges, resulting in increased dependence on complex mechanisms for automated test case selection and prioritisation as part of a continuous integration framework. These mechanisms are currently using simple entities called test cases that are concretely realised as executable scripts. Our key idea is to provide test cases with more reasoning, adaptive behaviour and learning capabilities by using the concepts of intelligent software agents. We refer to such test cases as test agents. The model that underlie a test agent is capable of flexible and autonomous actions in order to meet overall testing objectives. Our goal is to increase the decentralisation of regression testing by letting test agents to know for themselves when they should be executing, how they should update their purpose, and when they should interact with each other. In this paper, we envision software test agents that display such adaptive autonomous behaviour. Emerging developments and challenges regarding the use of test agents are explored-in particular, new research that seeks to use adaptive autonomous agents in software testing.
|
|
| 42. |
- Enoiu, Eduard Paul, PhD, et al.
(författare)
-
Test agents : The next generation of test cases
- 2019
-
Ingår i: Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2019. - : Institute of Electrical and Electronics Engineers Inc.. - 9781728108889 ; , s. 305-308
-
Konferensbidrag (refereegranskat)abstract
- Growth of software size, lack of resources to perform regression testing, and failure to detect bugs faster have seen increased reliance on continuous integration and test automation. Even with greater hardware and software resources dedicated to test automation, software testing is faced with enormous challenges, resulting in increased dependence on centralized and complex mechanisms for automated test case selection as part of continuous integration. These mechanisms are currently using static entities called test cases that are concretely realized as executable scripts. Our key vision is to provide test cases with more reasoning, adaptive behavior and learning capabilities by using the concepts of software agents. We refer to such test cases as test agents. The model that underlie a test agent is capable of flexible and autonomous actions in order to meet overall testing objectives. Our goal is to increase the decentralization of regression testing by letting test agents to know for themselves when they should be executing, how they should update their purpose, and when they should interact with each other. In this paper, we envision test agents that display such adaptive autonomous behavior. Existing and emerging developments and challenges regarding the use of test agents are explored - in particular, new research that seeks to use adaptive autonomous agents in software testing.
|
|
| 43. |
- Enoiu, Eduard Paul, PhD, et al.
(författare)
-
Towards a Model of Testers' Cognitive Processes: Software Testing as a Problem Solving Approach
- 2020
-
Ingår i: Proceedings - Companion of the 2020 IEEE 20th International Conference on Software Quality, Reliability, and Security, QRS-C 2020. ; , s. 272-279
-
Konferensbidrag (refereegranskat)abstract
- Software testing is a complex, intellectual activity based (at least) on analysis, reasoning, decision making, abstraction and collaboration performed in a highly demanding environment. Naturally, it uses and allocates multiple cognitive resources in software testers. However, while a cognitive psychology perspective is increasingly used in the general software engineering literature, it has yet to find its place in software testing. To the best of our knowledge, no theory of software testers' cognitive processes exists. Here, we take the first step towards such a theory by presenting a cognitive model of software testing based on how problem solving is conceptualized in cognitive psychology. Our approach is to instantiate a general problem solving process for the specific problem of creating test cases. We then propose an experiment for testing our cognitive test design model. The experiment makes use of verbal protocol analysis to understand the mechanisms by which human testers choose, design, implement and evaluate test cases. An initial evaluation was then performed with five software engineering master students as subjects. The results support a problem solving-based model of test design for capturing testers' cognitive processes.
|
|
| 44. |
- Enoiu, Eduard Paul, PhD, et al.
(författare)
-
Towards Human-Like Automated Test Generation: Perspectives from Cognition and Problem Solving
- 2021
-
Ingår i: Proceedings - 2021 IEEE/ACM 13th International Workshop on Cooperative and Human Aspects of Software Engineering, CHASE 2021. - Virtual and Madrid, Spain. ; , s. 123-124
-
Konferensbidrag (refereegranskat)abstract
- Automated testing tools typically create test cases that are different from what human testers create. This often makes the tools less effective, the created tests harder to understand, and thus results in tools providing less support to human testers. Here, we propose a framework based on cognitive science and, in particular, an analysis of approaches to problem solving, for identifying cognitive processes of testers. The framework helps map test design steps and criteria used in human test activities and thus to better understand how effective human testers perform their tasks. Ultimately, our goal is to be able to mimic how humans create test cases and thus to design more human-like automated test generation systems. We posit that such systems can better augment and support testers in a way that is meaningful to them.
|
|
| 45. |
- Enoiu, Eduard Paul, PhD, et al.
(författare)
-
Understanding Problem Solving in Software Testing : An Exploration of Tester Routines and Behavior
- 2023
-
Ingår i: Lecture Notes Computer Science. - : Springer Science and Business Media Deutschland GmbH. - 9783031432392 ; 14131 LNCS, s. 143-159
-
Konferensbidrag (refereegranskat)abstract
- Software testing is a difficult, intellectual activity performed in a social environment. Naturally, testers use and allocate multiple cognitive resources towards this task. The goal of this study is to understand better the routine and behaviour of human testers and their mental models when performing testing. We investigate this topic by surveying 38 software testers and developers in Sweden. The survey explores testers’ cognitive processes when performing testing by investigating the knowledge they bring, the activities they select and perform, and the challenges they face in their routine. By analyzing the survey results, we provide a characterization of tester practices and identify insights regarding the problem-solving process. We use these descriptions to further enhance a cognitive model of software testing. © 2023, IFIP International Federation for Information Processing.
|
|
| 46. |
- Enoiu, Eduard Paul, PhD, et al.
(författare)
-
VeriDevOps Software Methodology : Security Verification and Validation for DevOps Practices
- 2023
-
Ingår i: ACM International Conference Proceeding Series. - : Association for Computing Machinery. - 9798400707728
-
Konferensbidrag (refereegranskat)abstract
- VeriDevOps offers a methodology and a set of integrated mechanisms that significantly improve automation in DevOps to protect systems at operations time and prevent security issues at development time by (1) specifying security requirements, (2) generating trace monitors, (3) locating root causes of vulnerabilities, and (4) identifying security flaws in code and designs. This paper presents a methodology that enhances productivity and enables the continuous integration/delivery of trustworthy systems. We outline the methodology, its application to relevant scenarios, and offer recommendations for engineers and managers adopting the VeriDevOps approach. Practitioners applying the VeriDevOps methodology should include security modeling in the DevOps process, integrate security verification throughout all stages, utilize automated test generation tools for security requirements, and implement a comprehensive security monitoring system, with regular review and update procedures to maintain relevance and effectiveness.
|
|
| 47. |
- Entekhabi, Sina, 1989-
(författare)
-
Test Automation for Grid-Based Multiagent Autonomous Systems
- 2024
-
Licentiatavhandling (övrigt vetenskapligt/konstnärligt)abstract
- Traditional software testing usually comes with manual definitions of test cases. This manual process can be time-consuming, tedious, and incomplete in covering important but elusive corner cases that are hardly identifiable. Automatic generation of random test cases emerges as a strategy to mitigate the challenges associated with the manual test case design. However, the effectiveness of random test cases in fault detection may be limited, leading to increased testing costs, particularly in systems where test execution demands substantial resources and time. Leveraging the domain knowledge of test experts can guide the automatic random generation of test cases to more effective zones. In this thesis, we target quality assurance of multiagent autonomous systems and aim to automate test generation for them by applying the domain knowledge of test experts.To formalize the specification of the domain expert's knowledge, we introduce a small Domain Specific Language (DSL) for formal specification of particular locality-based constraints for grid-based multiagent systems. We initially employ this DSL for filtering randomly generated test inputs. Then, we evaluate the effectiveness of the generated test cases through an experiment on a case study of autonomous agents. Applying statistical analysis on the experiment results demonstrates that utilizing the domain knowledge to specify test selection criteria for filtering randomly generated test cases significantly reduces the number of potentially costly test executions to identify the persisting faults. Domain knowledge of experts can also be utilized to directly generate test inputs with constraint solvers. We conduct a comprehensive study to compare the performance of filtering random cases and constraint-solving approaches in generating selective test cases across various test scenario parameters. The examination of these parameters provides criteria for determining the suitability of random data filtering versus constraint solving, considering the varying size and complexity of the test input generation constraint. To conduct our experiments, we use QuickCheck tool for random test data generation with filtering, and we employ Z3 for constraint solving. The findings, supported by observations and statistical analysis, reveal that test scenario parameters impact the performance of filtering and constraint-solving approaches differently. Specifically, the results indicate complementary strengths between the two approaches: random generation and filtering approach excels for the systems with a large number of agents and long agent paths but shows degradation in larger grid sizes and stricter constraints. Conversely, constraint solving approach demonstrates robust performance for large grid sizes and strict constraints but experiences degradation with increased agent numbers and longer paths.Our initially proposed DSL is limited in its features and is only capable of specifying particular locality-based constraints. To be able to specify more elaborate test scenarios, we extend that DSL based on a more intricate model of autonomous agents and their environment. Using the extended DSL, we can specify test oracles and test scenarios for a dynamic grid environment and agents having several attributes. To assess the extended DSL's utility, we design a questionnaire to gather opinions from several experts and also run an experiment to compare the efficiency of the extended DSL with the initially proposed one. The questionnaire results indicate that the extended DSL was successful in specifying several scenarios that the experts found more useful than the scenarios specified by the initial DSL. Moreover, the experimental results demonstrate that testing with the extended DSL can significantly reduce the number of test executions to detect system faults, leading to a more efficient testing process.
|
|
| 48. |
- Ericsson, Sara, et al.
(författare)
-
Combinatorial modeling and test case generation for industrial control software using ACTS
- 2018
-
Ingår i: Proceedings - 2018 IEEE 18th International Conference on Software Quality, Reliability, and Security, QRS 2018. - : Institute of Electrical and Electronics Engineers Inc.. - 9781538677575 ; , s. 414-425
-
Konferensbidrag (refereegranskat)abstract
- Combinatorial testing has been suggested as an effective method of creating test cases at a lower cost. However, industrially applicable tools for modeling and combinatorial test generation are still scarce. As a direct effect, combinatorial testing has only seen a limited uptake in industry that calls into question its practical usefulness. This lack of evidence is especially troublesome if we consider the use of combinatorial test generation for industrial safety-critical control software, such as are found in trains, airplanes, and power plants. To study the industrial application of combinatorial testing, we evaluated ACTS, a popular tool for combinatorial modeling and test generation, in terms of applicability and test efficiency on industrial-sized IEC 61131-3 industrial control software running on Programmable Logic Controllers (PLC). We assessed ACTS in terms of its direct applicability in combinatorial modeling of IEC 61131-3 industrial software and the efficiency of ACTS in terms of generation time and test suite size. We used 17 industrial control programs provided by Bombardier Transportation Sweden AB and used in a train control management system. Our results show that not all combinations of algorithms and interaction strengths could generate a test suite within a realistic cut-off time. The results of the modeling process and the efficiency evaluation of ACTS are useful for practitioners considering to use combinatorial testing for industrial control software as well as for researchers trying to improve the use of such combinatorial testing techniques.
|
|
| 49. |
- Felderer, Michael, 1978-, et al.
(författare)
-
Artificial Intelligence Techniques in System Testing
- 2023
-
Ingår i: Optimising the software development process with artificial intelligence. - : Springer Science and Business Media Deutschland GmbH. - 9789811999475 - 9789811999482 ; , s. 221-240
-
Bokkapitel (övrigt vetenskapligt/konstnärligt)abstract
- System testing is essential for developing high-quality systems, but the degree of automation in system testing is still low. Therefore, there is high potential for Artificial Intelligence (AI) techniques like machine learning, natural language processing, or search-based optimization to improve the effectiveness and efficiency of system testing. This chapter presents where and how AI techniques can be applied to automate and optimize system testing activities. First, we identified different system testing activities (i.e., test planning and analysis, test design, test execution, and test evaluation) and indicated how AI techniques could be applied to automate and optimize these activities. Furthermore, we presented an industrial case study on test case analysis, where AI techniques are applied to encode and group natural language into clusters of similar test cases for cluster-based test optimization. Finally, we discuss the levels of autonomy of AI in system testing.
|
|
| 50. |
- Fifo, Miraldi, et al.
(författare)
-
On Measuring Combinatorial Coverage of Manually Created Test Cases for Industrial Software
- 2019
-
Ingår i: International Conference on Software Testing, Verification and Validation Workshops ICSTW19. ; , s. 264-267
-
Konferensbidrag (refereegranskat)abstract
- Combinatorial coverage has been proposed as a way to measure the quality of test cases by using the input interaction characteristics. This paper describes the results of empirically measuring combinatorial coverage of manually created test cases by experienced industrial engineers. We found that manual test cases achieve on average 78% 2-way combinatorial coverage, 57% 3-way coverage, 40% 4-way coverage, 20% 5-way combinatorial coverage and 13% for 6-way combinatorial coverage. These manual test cases can be augmented to achieve 100% combinatorial coverage for 2-way and 3-way interactions by adding eight and 66 missing tests on average, respectively. For 4-way interactions, full combinatorial coverage can achieved by adding 658 missing tests. For 5-way and 6-way interactions, full combinatorial coverage can be achieved by adding 5163 and 6170 missing tests on average, respectively. The results of this paper suggest that manual tests created by industrial engineers do no achieve high combinatorial coverage and can be improved by using combinatorial testing at the expense of the number of test cases to be executed.
|
|
