| 1. |
- Brahneborg, Daniel, 1970-, et al.
(författare)
-
Round-Trip Time Anomaly Detection
- 2018
-
Ingår i: ICPE '18 Proceedings of the 2018 ACM/SPEC International Conference on Performance Engineering. - New York, NY, USA : ACM. - 9781450350952 ; , s. 107-114
-
Konferensbidrag (refereegranskat)abstract
- Mobile text messages (SMS) are sometimes used for authentication, which requires short and reliable delivery times. The observed round-trip times when sending an SMS message provide valuable information on the quality of the connection. In this industry paper, we propose a method for detecting round-trip time anomalies, where the exact distribution is unknown, the variance is several orders of magnitude, and there are lots of shorter spikes that should be ignored. In particular, we show that using an adaption of Double Seasonal Exponential Smoothing to reduce the content dependent variations, followed by the Remedian to find short-term and long-term medians, successfully identifies larger groups of outliers. As training data for our method we use log files from a live SMS gateway. In order to verify the effectiveness of our approach, we utilize simulated data. Our contributions are a description on how to isolate content dependent variations, and the sequence of steps to find significant anomalies in big data.
|
|
| 2. |
- Flemström, Daniel, et al.
(författare)
-
A Research Roadmap for Test Design in Automated Integration Testing of Vehicular Systems
- 2016. - 9
-
Ingår i: FASSI 2016. - : International Academy, Research and Industry Association (IARIA). - 9781612084978 ; , s. 18-23
-
Konferensbidrag (refereegranskat)abstract
- An increasing share of the innovations emerging in the vehicular industry are implemented in software. Consequently, vehicular electrical systems are becoming more and more complex with an increasing number of functions, computational nodes and complex sensors, e.g., cameras and radars. The introduction of autonomous functional components, such as advanced driver assistance systems, highlight the foreseeable complexity of different parts of the system interacting with each other and with the human driver. It is of utmost importance that the testing effort can scale with this increasing complexity. In this paper, we review the challenges that we are facing in integration testing of complex embedded vehicular systems. Further, based on these challenges we outline a set of research directions for semi-automated or automated test design and execution in integration testing of vehicular systems. While the discussion is exemplified with our hands-on experience of the automotive industry, much of the concepts can be generalised to a broader setting of complex embedded systems.
|
|
| 3. |
- Flemström, Daniel, et al.
(författare)
-
Exploring Test Overlap in System Integration : An Industrial Case Study
- 2016
-
Ingår i: 42nd Euromicro Conference series on Software Engineering and Advanced Applications SEAA 2016. - 9781509028191 ; , s. 303-312
-
Konferensbidrag (refereegranskat)abstract
- Tougher safety regulations, global competition and ever increasing complexity of embedded software puts extensive pressure on the effectiveness of the software testing procedures. Previous studies have found that there exist overlaps (i.e., multiple instances of highly similar test cases) and even redundancies in the software testing process. Such overlap has been found between versions, variants and integration levels, but primarily at unit test level. Given large embedded systems involving many subsystems, does overlap exist within the system integration testing as well? In this paper, we present an industrial case study, aiming to a) evaluate if there exist test overlaps within the given context, b) if so, investigate how these overlaps are distributed, and c) find ways of reducing test effort by investigating how the knowledge of overlaps and their distribution may be used for finding candidate test cases for automation, maintenance or even removal. We have studied manual test cases, written in natural language, at a large vehicular manufacturer in Sweden. In particular, we have collected and analyzed test cases from the system integration testing levels of four different projects of a vehicle control management system. Using a similarity function, we evaluate if any overlaps between test cases exist, and where. We found that overlaps do exist within the system integration level, particularly in the form of partial test step sequences. However, very few test cases overlapped in their entirety. Some candidates for test step automation and update propagation were identified, but none for easy removal.
|
|
| 4. |
- Flemström, Daniel, et al.
(författare)
-
From natural language requirements to passive test cases using guarded assertions
- 2018
-
Ingår i: Proceedings - 2018 IEEE 18th International Conference on Software Quality, Reliability, and Security, QRS 2018. - : Institute of Electrical and Electronics Engineers Inc.. - 9781538677575 ; , s. 470-481
-
Konferensbidrag (refereegranskat)abstract
- In large-scale embedded system development, requirements are often expressed in natural language. Translating these requirements to executable test cases, while keeping the test cases and requirements aligned, is a challenging task. While such a transformation typically requires extensive domain knowledge, we show that a systematic process in combination with passive testing would facilitate the translation as well as linking the requirements to tests. Passive testing approaches observe the behavior of the system and test their correctness without interfering with the normal behavior. We use a specific approach to passive testing: guarded assertions (G/A). This paper presents a method for transforming system requirements expressed in natural language into G/As. We further present a proof of concept evaluation, performed at Bombardier Transportation Sweden AB, in which we show how the process would be used, together with practical advice of the reasoning behind the translation steps.
|
|
| 5. |
- Flemström, Daniel, et al.
(författare)
-
Similarity-based prioritization of test case automation
- 2018
-
Ingår i: Software quality journal. - : Springer Science and Business Media LLC. - 0963-9314 .- 1573-1367. ; 26:4, s. 1421-1449
-
Tidskriftsartikel (refereegranskat)abstract
- The importance of efficient software testing procedures is driven by an ever increasing system complexity as well as global competition. In the particular case of manual test cases at the system integration level, where thousands of test cases may be executed before release, time must be well spent in order to test the system as completely and as efficiently as possible. Automating a subset of the manual test cases, i.e, translating the manual instructions to automatically executable code, is one way of decreasing the test effort. It is further common that test cases exhibit similarities, which can be exploited through reuse when automating a test suite. In this paper, we investigate the potential for reducing test effort by ordering the test cases before such automation, given that we can reuse already automated parts of test cases. In our analysis, we investigate several approaches for prioritization in a case study at a large Swedish vehicular manufacturer. The study analyzes the effects with respect to test effort, on four projects with a total of 3919 integration test cases constituting 35,180 test steps, written in natural language. The results show that for the four projects considered, the difference in expected manual effort between the best and the worst order found is on average 12 percentage points. The results also show that our proposed prioritization method is nearly as good as more resource demanding meta-heuristic approaches at a fraction of the computational time. Based on our results, we conclude that the order of automation is important when the set of test cases contain similar steps (instructions) that cannot be removed, but are possible to reuse. More precisely, the order is important with respect to how quickly the manual test execution effort decreases for a set of test cases that are being automated.
|
|
| 6. |
- Flemström, Daniel, 1971-
(författare)
-
Similarity-Based Test Effort Reduction
- 2017
-
Licentiatavhandling (övrigt vetenskapligt/konstnärligt)abstract
- Embedded computer systems are all around us. We find them in everything, from dishwashers to cars and airplanes. They must always work correctly and moreover, often within certain time constraints. The software of such a system can be very large and complex, e.g. in the case of a car or a train. Hence, we develop the software for embedded systems in smaller, manageable, parts. These parts can be successively integrated, until they form the complete software for the embedded system, possibly at different levels. This phase of the development process is called the system integration phase and is one of the most critical phases in the development of embedded systems. In this phase, substantial effort is spent on testing activities.Studies have found that a considerable amount of test effort is wasteful due to people, unknowingly or by necessity, performing similar (or even overlapping) test activities. Consequently, test cases may end up to be similar, partially or wholly. We identified such test similarities in a case study of 2500 test cases, written in natural language, from four different projects in the embedded vehicular domain. Such information can be used for reducing effort when maintaining or automating similar test cases.In another case study in the same domain, we investigated several approaches for prioritizing test cases to automate with the objective to reduce manual test effort as quick as possible given that similar automated tests could be reused (similarity-based reuse). We analyzed how the automation order affects the test effort for four projects with a total of 3919 integration test cases, written in natural language. The results showed that similarity-based reuse of automated test case script code, and the best-performing automation order can reduce the expected manual test effort with 20 percentage points.Another way of reducing test effort is to reuse test artifacts from one level of integration to another, instead of duplicating them. We studied such reuse methods, that we denote vertical reuse, in a systematic mapping study. While the results from of our systematic mapping study showed the viability of vertical test reuse methods, our industrial case studies showed that keeping track of similarities and test overlaps is both possible and feasible for test effort reduction. We further conclude that the test case automation order affects the manual test execution effort when there exist similar steps that cannot be removed, but are possible to reuse with respect to test script code.
|
|
| 7. |
- Flemström, Daniel, et al.
(författare)
-
Vertical Test Reuse for Embedded Systems : A Systematic Mapping Study
- 2015. - 11
-
Ingår i: 2015 41st Euromicro Conference on Software Engineering and Advanced Applications. - : Conference Publishing Services. - 9781467375856 ; , s. 317-324
-
Konferensbidrag (refereegranskat)abstract
- Vertical test reuse refers to the the reuse of test cases or other test artifacts over different integration levels in the software or system engineering process. Vertical test reuse has previously been proposed for reducing test effort and improving test effectiveness, particularly for embedded system development. The goal of this study is to provide an overview of the state of the art in the field of vertical test reuse for embedded system development. For this purpose, a systematic mapping study has been performed, identifying 11 papers on vertical test reuse for embedded systems. The primary result from the mapping is a classification of published work on vertical test reuse in the embedded system domain, covering motivations for reuse, reuse techniques, test levels and reusable test artifacts considered, and to what extent the effects of reuse have been evaluated.
|
|
| 8. |
- Abbas, Muhammad, et al.
(författare)
-
MBRP : Model-Based Requirements Prioritization Using PageRank Algorithm
- 2019
-
Ingår i: 2019 26th Asia-Pacific Software Engineering Conference (APSEC). - : IEEE conference proceedings. - 9781728146485 ; , s. 31-38
-
Konferensbidrag (refereegranskat)abstract
- Requirements prioritization plays an important role in driving project success during software development. Literature reveals that existing requirements prioritization approaches ignore vital factors such as interdependency between requirements. Existing requirements prioritization approaches are also generally time-consuming and involve substantial manual effort. Besides, these approaches show substantial limitations in terms of the number of requirements under consideration. There is some evidence suggesting that models could have a useful role in the analysis of requirements interdependency and their visualization, contributing towards the improvement of the overall requirements prioritization process. However, to date, just a handful of studies are focused on model-based strategies for requirements prioritization, considering only conflict-free functional requirements. This paper uses a meta-model-based approach to help the requirements analyst to model the requirements, stakeholders, and inter-dependencies between requirements. The model instance is then processed by our modified PageRank algorithm to prioritize the given requirements. An experiment was conducted, comparing our modified PageRank algorithm's efficiency and accuracy with five existing requirements prioritization methods. Besides, we also compared our results with a baseline prioritized list of 104 requirements prepared by 28 graduate students. Our results show that our modified PageRank algorithm was able to prioritize the requirements more effectively and efficiently than the other prioritization methods.
|
|
| 9. |
- Abbaspour Asadollah, Sara, et al.
(författare)
-
10 Years of research on debugging concurrent and multicore software : a systematic mapping study
- 2017
-
Ingår i: Software quality journal. - : Springer Science and Business Media LLC. - 0963-9314 .- 1573-1367. ; 25:1, s. 49-82
-
Tidskriftsartikel (refereegranskat)abstract
- Debugging – the process of identifying, localizing and fixing bugs – is a key activity in software development. Due to issues such as non-determinism and difficulties of reproducing failures, debugging concurrent software is significantly more challenging than debugging sequential software. A number of methods, models and tools for debugging concurrent and multicore software have been proposed, but the body of work partially lacks a common terminology and a more recent view of the problems to solve. This suggests the need for a classification, and an up-to-date comprehensive overview of the area. This paper presents the results of a systematic mapping study in the field of debugging of concurrent and multicore software in the last decade (2005– 2014). The study is guided by two objectives: (1) to summarize the recent publication trends and (2) to clarify current research gaps in the field.Through a multi-stage selection process, we identified 145 relevant papers. Based on these, we summarize the publication trend in the field by showing distribution of publications with respect to year , publication venues , representation of academia and industry , and active research institutes . We also identify research gaps in the field based on attributes such as types of concurrency bugs, types of debugging processes , types of research and research contributions.The main observations from the study are that during the years 2005–2014: (1) there is no focal conference or venue to publish papers in this area, hence a large variety of conferences and journal venues (90) are used to publish relevant papers in this area; (2) in terms of publication contribution, academia was more active in this area than industry; (3) most publications in the field address the data race bug; (4) bug identification is the most common stage of debugging addressed by articles in the period; (5) there are six types of research approaches found, with solution proposals being the most common one; and (6) the published papers essentially focus on four different types of contributions, with ”methods” being the type most common one.We can further conclude that there is still quite a number of aspects that are not sufficiently covered in the field, most notably including (1) exploring correction and fixing bugs in terms of debugging process; (2) order violation, suspension and starvation in terms of concurrency bugs; (3) validation and evaluation research in the matter of research type; (4) metric in terms of research contribution. It is clear that the concurrent, parallel and multicore software community needs broader studies in debugging.This systematic mapping study can help direct such efforts.
|
|
| 10. |
- Abbaspour Asadollah, Sara, et al.
(författare)
-
A Model for Systematic Monitoring and Debugging of Starvation Bugs in Multicore Software
- 2016
-
Ingår i: Proceedings of the 1st International Workshop on Specification, Comprehension, Testing, and Debugging of Concurrent Programs (SCTDCP 2016). - New York, NY, USA : ACM. - 9781450345101 ; , s. 7-11
-
Konferensbidrag (refereegranskat)abstract
- With the development of multicore hardware, concurrent, parallel and multicore software are becoming increasingly popular. Software companies are spending a huge amount of time and resources to nd and debug the bugs. Among all types of software bugs, concurrency bugs are also important and troublesome. This type of bugs is increasingly becoming an issue particularly due to the growing prevalence of multicore hardware. In this position paper, we propose a model for monitoring and debugging Starvation bugs as a type of concurrency bugs in multicore software. The model is composed into three phases: monitoring, detecting and debugging. The monitoring phase can support detecting phase by storing collected data from the system execution. The detecting phase can support debugging phase by comparing the stored data with starvation bug's properties, and the debugging phase can help in reproducing and removing the Starvation bug from multicore software. Our intention is that our model is the basis for developing tool(s) to enable solving Starvation bugs in software for multicore platforms.
|
|
| 11. |
|
|
| 12. |
- Abbaspour Asadollah, Sara, et al.
(författare)
-
A Runtime Verification Tool for Detecting Concurrency Bugs in FreeRTOS Embedded Software
- 2018
-
Ingår i: Proceedings - 17th International Symposium on Parallel and Distributed Computing, ISPDC 2018. - : Institute of Electrical and Electronics Engineers Inc.. - 9781538653302 ; , s. 172-179
-
Konferensbidrag (refereegranskat)abstract
- This article presents a runtime verification tool for embedded software executing under the open source real-time operating system FreeRTOS. The tool detects and diagnoses concurrency bugs such as deadlock, starvation, and suspension based-locking. The tool finds concurrency bugs at runtime without debugging and tracing the source code. The tool uses the Tracealyzer tool for logging relevant events. Analysing the logs, our tool can detect the concurrency bugs by applying algorithms for diagnosing each concurrency bug type individually. In this paper, we present the implementation of the tool, as well as its functional architecture, together with illustration of its use. The tool can be used during program testing to gain interesting information about embedded software executions. We present initial results of running the tool on some classical bug examples running on an AVR 32-bit board SAM4S.
|
|
| 13. |
- Abbaspour Asadollah, Sara, et al.
(författare)
-
A Study on Concurrency Bugs in an Open Source Software
- 2016
-
Ingår i: IFIP Advances in Information and Communication Technology, vol. 472. - Cham : Springer International Publishing. - 9783319392240 ; , s. 16-31
-
Konferensbidrag (refereegranskat)abstract
- Concurrent programming puts demands on software debugging and testing, as concurrent software may exhibit problems not present in sequential software, e.g., deadlocks and race conditions. In aiming to increase efficiency and effectiveness of debugging and bug-fixing for concurrent software, a deep understanding of concurrency bugs, their frequency and fixingtimes would be helpful. Similarly, to design effective tools and techniques for testing and debugging concurrent software understanding the differences between non-concurrency and concurrency bugs in real-word software would be useful.
|
|
| 14. |
|
|
| 15. |
- Abbaspour Asadollah, Sara, et al.
(författare)
-
Concurrency bugs in open source software : a case study
- 2017
-
Ingår i: Journal of Internet Services and Applications. - : Springer London. - 1867-4828 .- 1869-0238. ; 8:1
-
Tidskriftsartikel (refereegranskat)abstract
- Concurrent programming puts demands on software debugging and testing, as concurrent software may exhibit problems not present in sequential software, e.g., deadlocks and race conditions. In aiming to increase efficiency and effectiveness of debugging and bug-fixing for concurrent software, a deep understanding of concurrency bugs, their frequency and fixing-times would be helpful. Similarly, to design effective tools and techniques for testing and debugging concurrent software, understanding the differences between non-concurrency and concurrency bugs in real-word software would be useful. This paper presents an empirical study focusing on understanding the differences and similarities between concurrency bugs and other bugs, as well as the differences among various concurrency bug types in terms of their severity and their fixing time, and reproducibility. Our basis is a comprehensive analysis of bug reports covering several generations of five open source software projects. The analysis involves a total of 11860 bug reports from the last decade, including 351 reports related to concurrency bugs. We found that concurrency bugs are different from other bugs in terms of their fixing time and severity while they are similar in terms of reproducibility. Our findings shed light on concurrency bugs and could thereby influence future design and development of concurrent software, their debugging and testing, as well as related tools.
|
|
| 16. |
- Abbaspour Asadollah, Sara, et al.
(författare)
-
Runtime Verification for Detecting Suspension Bugs in Multicore and Parallel Software
- 2017
-
Ingår i: Proceedings - 10th IEEE International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2017. - 9781509066766 ; , s. 77-80
-
Konferensbidrag (refereegranskat)abstract
- Multicore hardware development increases the popularity of parallel and multicore software, while testing and debugging the software become more difficult, frustrating and costly. Among all types of software bugs, concurrency bugs are both important and troublesome. This type of bugs is increasingly becoming an issue, particularly due to the growing prevalence of multicore hardware. Suspension-based-locking bug is one type of concurrency bugs. This position paper proposes a model based on runtime verification and reflection technique in the context of multicore and parallel software to monitor and detect suspension-based-locking bugs. The model is not only able to detect faults, but also diagnose and even repair them. The model is composed of four layers: Logging, Monitoring, Suspension Bug Diagnosis and Mitigation. The logging layer will observe the events and save them into a file system. The monitoring layer will detect the presents of bugs in the software. The suspension bug diagnosis will identify Suspension bugs by comparing the captured data with the suspension bug properties. Finally, the mitigation layer will reconfigure the software to mitigate the suspension bugs. A functional architecture of a runtime verification tool is also proposed in this paper. This architecture is based on the proposed model and is comprised of different modules.
|
|
| 17. |
- Abbaspour Asadollah, Sara, et al.
(författare)
-
Towards Classification of Concurrency Bugs Based on Observable Properties
- 2015
-
Ingår i: Proceedings - 1st International Workshop on Complex Faults and Failures in Large Software Systems, COUFLESS 2015. - 9781479919345 ; , s. 41-47
-
Konferensbidrag (refereegranskat)abstract
- In software engineering, classification is a way to find an organized structure of knowledge about objects. Classification serves to investigate the relationship between the items to be classified, and can be used to identify the current gaps in the field. In many cases users are able to order and relate objects by fitting them in a category. This paper presents initial work on a taxonomy for classification of errors (bugs) related to concurrent execution of application level software threads. By classifying concurrency bugs based on their corresponding observable properties, this research aims to examine and structure the state of the art in this field, as well as to provide practitioner support for testing and debugging of concurrent software. We also show how the proposed classification, and the different classes of bugs, relates to the state of the art in the field by providing a mapping of the classification to a number of recently published papers in the software engineering field.
|
|
| 18. |
- Bilic, Damir, et al.
(författare)
-
Model-Based Product Line Engineering in an Industrial Automotive Context: An Exploratory Case Study
- 2018
-
Ingår i: 1st Intl. Workshop on Variability and Evolution of Software-intensive Systems VariVolution'18. - New York, NY, USA : ACM. ; , s. 56-63
-
Konferensbidrag (refereegranskat)abstract
- Product Line Engineering is an approach to reuse assets of complex systems by taking advantage of commonalities between product families. Reuse within complex systems usually means reuse of artifacts from different engineering domains such as mechanical, electronics and software engineering. Model-based systems engineering is becoming a standard for systems engineering and collaboration within different domains. This paper presents an exploratory case study on initial efforts of adopting Product Line Engineering practices within the model-based systems engineering process at Volvo Construction Equipment (Volvo CE), Sweden. We have used SysML to create overloaded models of the engine systems at Volvo CE. The variability within the engine systems was captured by using the Orthogonal Variability Modeling language. The case study has shown us that overloaded SysML models tend to become complex even on small scale systems, which in turn makes scalability of the approach a major challenge. For successful reuse and to, possibly, tackle scalability, it is necessary to have a database of reusable assets from which product variants can be derived.
|
|
| 19. |
- Derehag, Jesper, et al.
(författare)
-
Transitioning Fault Prediction Models to a New Environment
- 2016
-
Ingår i: Proceedings - 2016 12th European Dependable Computing Conference, EDCC 2016. - 9781509015825 ; , s. 241-248
-
Konferensbidrag (refereegranskat)abstract
- We describe the application and evaluation of fault prediction algorithms to a project developed by a Swedish company that transitioned from waterfall to agile development methods. The project used two different version control systems and a separate bug tracking system during its lifetime. The algorithms were originally designed for use on systems implemented with a traditional waterfall process at American companies that maintained their project records in an integrated database system that combined bug recording and version control. We compare the performance of the original prediction model on the American systems to the results obtained in the Swedish environment in both its pre-agile and agile stages. We also consider the impact of additional variables in the model.
|
|
| 20. |
- Enoiu, Eduard, 1986-
(författare)
-
Automatic test generation for industrial control software
- 2016
-
Doktorsavhandling (övrigt vetenskapligt/konstnärligt)abstract
- Since the early days of software testing, automatic test generation has been suggested as a way of allowing tests to be created at a lower cost. However, industrially useful and applicable tools for automatic test generation are still scarce. As a consequence, the evidence regarding the applicability or feasibility of automatic test generation in industrial practice is limited. This is especially problematic if we consider the use of automatic test generation for industrial safety-critical control systems, such as are found in power plants, airplanes, or trains.In this thesis, we improve the current state of automatic test generation by developing a technique based on model-checking that works with IEC 61131-3 industrial control software. We show how automatic test generation for IEC 61131-3 programs, containing both functional and timing information, can be solved as a model checking problem for both code and mutation coverage criteria. The developed technique has been implemented in the CompleteTest tool. To evaluate the potential application of our technique, we present several studies where the tool is applied to industrial control software. Results show that CompleteTest is viable for use in industrial practice; it is efficient in terms of the time required to generate tests that satisfy both code and mutation coverage and scales well for most of the industrial programs considered.However, our results also show that there are still challenges associated with the use of automatic test generation. In particular, we found that while automatically generated tests, based on code coverage, can exercise the logic of the software as well as tests written manually, and can do so in a fraction of the time, they do not show better fault detection compared to manually created tests. Specifically, it seems that manually created tests are able to detect more faults of certain types (i.e, logical replacement, negation insertion and timer replacement) than automatically generated tests. To tackle this issue, we propose an approach for improving fault detection by using mutation coverage as a test criterion. We implemented this approach in the CompleteTest tool and used it to evaluate automatic test generation based on mutation testing. While the resulting tests were more effective than automatic tests generated based on code coverage, in terms of fault detection, they still were not better than manually created tests.In summary, our results highlight the need for improving the goals used by automatic test generation tools. Specifically, fault detection scores could be increased by considering some new mutation operators as well as higher-order mutations. Our thesis suggests that automatically generated test suites are significantly less costly in terms of testing time than manually created test suites. One conclusion, strongly supported by the results of this thesis, is that automatic test generation is efficient but currently not quite as effective as manual testing. This is a significant progress that needs to be further studied; we need to consider the implications and the extent to which automatic test generation can be used in the development of reliable safety-critical systems.
|
|
| 21. |
- Enoiu, Eduard Paul, et al.
(författare)
-
A Comparative Study of Manual and Automated Testing for Industrial Control Software
- 2017
-
Ingår i: Proceedings - 10th IEEE International Conference on Software Testing, Verification and Validation, ICST 2017. - 9781509060313 ; , s. 412-417
-
Konferensbidrag (refereegranskat)abstract
- Automated test generation has been suggested as a way of creating tests at a lower cost. Nonetheless, it is not very well studied how such tests compare to manually written ones in terms of cost and effectiveness. This is particularly true for industrial control software, where strict requirements on both specification-based testing and code coverage typically are met with rigorous manual testing. To address this issue, we conducted a case study in which we compared manually and automatically created tests. We used recently developed real-world industrial programs written in the IEC 61131-3, a popular programming language for developing industrial control systems using programmable logic controllers. The results show that automatically generated tests achieve similar code coverage as manually created tests, but in a fraction of the time (an average improvement of roughly 90%). We also found that the use of an automated test generation tool does not result in better fault detection in terms of mutation score compared to manual testing. Specifically, manual tests more effectively detect logical, timer and negation type of faults, compared to automatically generated tests. The results underscore the need to further study how manual testing is performed in industrial practice and the extent to which automated test generation can be used in the development of reliable systems.
|
|
| 22. |
- Enoiu, Eduard Paul, et al.
(författare)
-
A Controlled Experiment in Testing of Safety-Critical Embedded Software
- 2016
-
Ingår i: Proceedings - 2016 IEEE International Conference on Software Testing, Verification and Validation, ICST 2016. - 9781509018260 ; , s. 1-11
-
Konferensbidrag (refereegranskat)abstract
- In engineering of safety critical systems, regulatory standards often put requirements on both traceable specification-based testing, and structural coverage on program units. Automated test generation techniques can be used to generate inputs to cover the structural aspects of a program. However, there is no conclusive evidence on how automated test generation compares to manual test design, or how testing based on the program implementation relates to specification-based testing. In this paper, we investigate specification-and implementation-based testing of embedded software written in the IEC 61131-3 language, a programming standard used in many embedded safety critical software systems. Further, we measure the efficiency and effectiveness in terms of fault detection. For this purpose, a controlled experiment was conducted, comparing tests created by a total of twenty-three software engineering master students. The participants worked individually on manually designing and automatically generating tests for two IEC 61131-3 programs. Tests created by the participants in the experiment were collected and analyzed in terms of mutation score, decision coverage, number of tests, and testing duration. We found that, when compared to implementation-based testing, specification-based testing yields significantly more effective tests in terms of the number of faults detected. Specifically, specification-based tests more effectively detect comparison and value replacement type of faults, compared to implementation-based tests. On the other hand, implementation-based automated test generation leads to fewer tests (up to 85% improvement) created in shorter time than the ones manually created based on the specification.
|
|
| 23. |
- Enoiu, Eduard Paul, et al.
(författare)
-
Automated Test Generation using Model-Checking: An Industrial Evaluation
- 2016
-
Ingår i: International Journal on Software Tools for Technology Transfer. - Germany : Springer. - 1433-2779 .- 1433-2787. ; 18:3, s. 335-353
-
Tidskriftsartikel (refereegranskat)abstract
- In software development, testers often focus on functional testing to validate implemented programs against their specifications. In safety critical software development, testers are also required to show that tests exercise, or cover, the structure and logic of the implementation. To achieve different types of logic coverage, various program artifacts such as decisions and conditions are required to be exercised during testing. Use of model-checking for structural test generation has been proposed by several researchers. The limited application to models used in practice and the state-space explosion can, however, impact model-checking and hence the process of deriving tests for logic coverage. Thus, there is a need to validate these approaches against relevant industrial systems such that more knowledge is built on how to efficiently use them in practice. In this paper, we present a tool-supported approach to handle software written in the Function Block Diagram language such that logic coverage criteria can be formalized and used by a model-checker to automatically generate tests. To this end, we conducted a study based on industrial use-case scenarios from Bombardier Transportation AB, showing how our toolbox COMPLETETEST can be applied to generate tests in software systems used in the safety-critical domain. To evaluate the approach, we applied the toolbox to 157 programs and found that it is efficient in terms of time required to generate tests that satisfy logic coverage and scales well for most of the programs.
|
|
| 24. |
- Enoiu, Eduard Paul, et al.
(författare)
-
Mutation-based test generation for PLC embedded software using model checking
- 2016
-
Ingår i: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). - Cham : Springer. - 9783319474427 ; , s. 155-171
-
Konferensbidrag (refereegranskat)abstract
- Testing is an important activity in engineering of industrial embedded software. In certain application domains (e.g., railway industry) engineering software is certified according to safety standards that require extensive software testing procedures to be applied for the development of reliable systems. Mutation analysis is a technique for creating faulty versions of a software for the purpose of examining the fault detection ability of a test suite. Mutation analysis has been used for evaluating existing test suites, but also for generating test suites that detect injected faults (i.e., mutation testing). To support developers in software testing, we propose a technique for producing test cases using an automated test generation approach that operates using mutation testing for software written in IEC 61131-3 language, a programming standard for safety-critical embedded software, commonly used for Programmable Logic Controllers (PLCs). This approach uses the Uppaal model checker and is based on a combined model that contains all the mutants and the original program. We applied this approach in a tool for testing industrial PLC programs and evaluated it in terms of cost and fault detection. For realistic validation we collected industrial experimental evidence on how mutation testing compares with manual testing as well as automated decision-coverage adequate test generation. In the evaluation, we used manually seeded faults provided by four industrial engineers. The results show that even if mutation-based test generation achieves better fault detection than automated decision coverage-based test generation, these mutation-adequate test suites are not better at detecting faults than manual test suites. However, the mutation-based test suites are significantly less costly to create, in terms of testing time, than manually created test suites. Our results suggest that the fault detection scores could be improved by considering some new and improved mutation operators (e.g., Feedback Loop Insertion Operator (FIO)) for PLC programs as well as higher-order mutations.
|
|
| 25. |
- Gustafsson, Thomas, et al.
(författare)
-
Automotive System Testing by Independent Guarded Assertions
- 2015. - 10
-
Ingår i: 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW). - 9781479918850
-
Konferensbidrag (refereegranskat)abstract
- Testing is a key activity in industry to verify and validate products before they reach end customers. In hardwarein- the-loop system-level verification of automotive systems, testing is often performed using sequential execution of test scripts, each containing a mix of stimuli and assertions. In this paper, we propose and study an alternative approach for automated system-level testing automotive systems. In our approach, assertion-only test scripts and one (or several) stimulionly script(s), execute concurrently on the test driver. By separating the stimuli from the assertions, with each assertion independently determining when the system under test shall be verified, we seek to achieve three things: 1) tests that better represent real-world handling of the product, 2) reduced test execution time, and 3) increased defect detection. In addition to describing our proposed approach in detail, we provide experimental results from an industrial case study evaluating the approach in an automotive system test environment.
|
|
| 26. |
- Hansson, Hans, et al.
(författare)
-
Real-Time in Networked Embedded Systems
- 2018
-
Ingår i: Embedded Systems Handbook: Second Edition. - : CRC Press. - 9781420074116 - 9781315218687 ; , s. 1-1-1-46
-
Bokkapitel (övrigt vetenskapligt/konstnärligt)
|
|
| 27. |
- Lindström, Birgitta, et al.
(författare)
-
Mutating Aspect-Oriented Models to Test Cross-Cutting Concerns
- 2015
-
Ingår i: 2015 IEEE 8th International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2015 - Proceedings. - : IEEE conference proceedings. - 9781479918850 ; , s. Article number 7107456-
-
Konferensbidrag (refereegranskat)
|
|
| 28. |
- Lindström, Birgitta, et al.
(författare)
-
Using mutation to design tests for aspect-oriented models
- 2017
-
Ingår i: Information and Software Technology. - : Elsevier BV. - 0950-5849 .- 1873-6025. ; 81, s. 112-130
-
Tidskriftsartikel (refereegranskat)abstract
- Context: Testing for properties such as robustness or security is complicated because their concerns are often repeated in many locations and muddled with the normal code. Such “cross-cutting concerns” include things like interrupt events, exception handling, and security protocols. Aspect-oriented (AO) modeling allows developers to model the cross-cutting behavior independently of the normal behavior, thus supporting model-based testing of cross-cutting concerns. However, mutation operators defined for AO programs (source code) are usually not applicable to AO models (AOMs) and operators defined for models do not target the AO features. Objective: We present a method to design abstract tests at the aspect-oriented model level. We define mutation operators for aspect-oriented models and evaluate the generated mutants for an example system. Method: AOMs are mutated with novel operators that specifically target the AO modeling features. Test traces killing these mutant models are then generated. The generated and selected traces are abstract tests that can be transformed to concrete black-box tests and run on the implementation level, to evaluate the behavior of the woven cross-cutting concerns (combined aspect and base models). Results: This paper is a significant extension of our paper at Mutation 2015. We present a complete fault model, additional mutation operators, and a thorough analysis of the mutants generated for an example system. Conclusions: The analysis shows that some mutants are stillborn (syntactically illegal) but none is equivalent (exhibiting the same behavior as the original model). Additionally, our AOM-specific mutation operators can be combined with pre-existing operators to mutate code or models without any overlap.
|
|
| 29. |
- Marinescu, Raluca, et al.
(författare)
-
Automatic Test Generation for Energy Consumption of Embedded Systems Modeled in EAST-ADL
- 2017
-
Ingår i: Proceedings - 10th IEEE International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2017. - 9781509066766 ; , s. 69-76
-
Konferensbidrag (refereegranskat)abstract
- Testing using architectural design models is intended to determine if the realized system meets its specification, and works as a whole in terms of computational components and their interactions. The growing complexity of embedded systems requires new techniques that are able to support testing of extra-functional requirements, like energy usage of components and systems, which is very necessary in order to obtain valid implementations. In this paper, we show how architectural models described in the EAST-ADL architectural language can also be used for testing the energy consumption of embedded systems, after transforming them into networks of formal models called priced timed automata. Assuming an EAST-ADL model annotated with energy consumption information, we show how to automatically generate energy-aware test cases based on statistical model checking (SMC) of the resulting network of priced timed automata. We automate the generation of executable test cases with UPPAAL SMC, using a test strategy based on several random simulation runs of the system. By seeding the original formal model with a set of energy-consumption related faults, we are able to carry out fault detection analysis. We apply this technique on a Brake-by-Wire system from the automotive domain, and evaluate it in terms of efficiency and model fault detection.
|
|
| 30. |
- Rodriguez-Navas, Guillermo, et al.
(författare)
-
Offline Analysis of Independent Guarded Assertions in Automotive Integration Testing
- 2015. - 7
-
Ingår i: 2015 IEEE 12th International Conference on Embedded Software and Systems. - 9781479989379 ; , s. 1066-1073
-
Konferensbidrag (refereegranskat)abstract
- The size and complexity of software in automotive systems have increased steadily over the last decades. Modern vehicles typically contain numerous electrical control units (ECUs), and more and more features require real-time interaction between several dedicated ECUs (e.g., gearbox, brake and engine control units) in order to perform their tasks. Since system safety and reliability must not be adversely affected by this increase in complexity, proper quality assurance is a must. Such quality assurance is often performed by testing the system in different levels of integration throughout the development process. However, the growth of complexity of the system under test aslo affects the testing, making it laborious, difficult and costly. This paper presents a novel method for efficient offline analysis of traces, which has been especially tailored for integration testing of automotive systems. The method exploits the recently defined concept of independent guarded assertion in order to formally describe the events that are relevant for the analysis as well as the expected behavior in those events. The offline analysis is implemented using a standard commercial model checker and has shown good performance in the conducted experiments.
|
|
| 31. |
- Shah, S. M. A., et al.
(författare)
-
Robustness Testing of Embedded Software Systems : An Industrial Interview Study
- 2016
-
Ingår i: IEEE Access. - : IEEE. - 2169-3536. ; 4, s. 1859-1871
-
Tidskriftsartikel (refereegranskat)abstract
- Embedded software is at the core of current and future telecommunication, automotive, multimedia, and industrial automation systems. The success of practically any industrial application depends on the embedded software system's dependability, and one method to verify the dependability of a system is testing its robustness. The motivation behind this paper is to provide a knowledge base of the state of the practice in robustness testing of embedded software systems and to compare this to the state of the art. We have gathered the information on the state of the practice in robustness testing from seven different industrial domains (telecommunication, automotive, multimedia, critical infrastructure, aerospace, consumer products, and banking) by conducting 13 semi-structured interviews. We investigate the different aspects of robustness testing, such as the general view of robustness, relation to requirements engineering and design, test execution, failures, and tools. We highlight knowledge from the state of the practice of robustness testing of embedded software systems. We found different robustness testing practices that have not been previously described. This paper shows that the state of the practice, when it comes to robustness testing, differs between organizations and is quite different from the state of the art described in the scientific literature. For example, methods commonly described in the literature (e.g., the fuzzy approach) are not used in the organizations we studied. Instead, the interviewees described several ad hoc approaches that take specific scenarios into account (e.g., power failure or overload). Other differences we found concern the classification of robustness failures, the hypothesized root causes of robustness failures, and the types of tools used for robustness testing. This paper is a first step in capturing the state of the practice of robustness testing of embedded software systems. The results can be used by both researchers and practitioners. Researchers can use our findings to understand the gap between the state of the art and the state of the practice and develop their studies to fill this gap. Practitioners can also learn from this knowledge base regarding how they can improve their practice and acquire other practices.
|
|
| 32. |
- Strandberg, Per, et al.
(författare)
-
Automated System Level Regression Test Prioritization in a Nutshell
- 2017
-
Ingår i: IEEE Software. - 0740-7459 .- 1937-4194. ; 34:4, s. 30-37
-
Tidskriftsartikel (refereegranskat)abstract
- Westermo Research and Development has developed SuiteBuilder, an automated tool to determine an effective ordering of regression test cases. The ordering is based on factors such as fault detection success, the interval since the last execution, and code modifications. SuiteBuilder has enabled Westermo to overcome numerous regression-testing problems, including lack of time to run a complete regression suite, failure to detect bugs in a timely manner, and repeatedly omitted tests. In the tool's first two years of use, reordered test suites finished in the available time, most fault-detecting test cases were located in the first third of suites, no important test case was omitted, and the necessity for manual work on the suites decreased greatly.
|
|
| 33. |
- Strandberg, Per Erik
(författare)
-
Automated System Level Software Testing of Networked Embedded Systems
- 2018
-
Licentiatavhandling (övrigt vetenskapligt/konstnärligt)abstract
- Embedded systems are ubiquitous and play critical roles in management systems for industry and transport. Software failures in these domains may lead to loss of production or even loss of life, so the software in these systems needs to be reliable. Software testing is a standard approach for quality assurance of embedded software, and many software development processes strive for test automation. However, important challenges for successful software test automation are: lack of time for testing, lack of test environment availability, and an excess of test results information that renders decision-making hard.In this thesis these challenges are tackled in three ways. First, in order to combat lack of time for testing, a method for automated system level regression test selection was implemented and evaluated using data from several years of nightly testing in a real world industrial setting. It was shown that automated test selection can be integrated into system level nightly testing and solve problems such as nightly testing not finishing on time. Second, in order to improve the hardware coverage of devices in the test environment and how test cases map to hardware, an algorithm based on the subgraph isomorphism problem was implemented and evaluated using industrial data. This implementation was significantly faster than the previous implementation, and the mapping process was done in such a way that hardware coverage increased over iterations. Third, to better understand decision-making in critical steps of the software development process in an industrial setting, two empirical studies were conducted. The results showed how visualizations and a test results database support decision-making. Results also describe the overall flow of information in software testing: from developers to hardware, and back to developers via the test results database.Automated system level software testing of networked embedded systems can be difficult to achieve. This thesis addresses several important challenges and provides results that are of interest both to industrial practitioners and researchers.
|
|
| 34. |
- Strandberg, Per Erik, et al.
(författare)
-
Automated test mapping and coverage for network topologies
- 2018
-
Ingår i: ISSTA 2018 - Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis. - New York, NY, USA : Association for Computing Machinery, Inc. - 9781450356992 ; , s. 73-83
-
Konferensbidrag (refereegranskat)abstract
- Communication devices such as routers and switches play a critical role in the reliable functioning of embedded system networks. Dozens of such devices may be part of an embedded system network, and they need to be tested in conjunction with various computational elements on actual hardware, in many different configurations that are representative of actual operating networks. An individual physical network topology can be used as the basis for a test system that can execute many test cases, by identifying the part of the physical network topology that corresponds to the configuration required by each individual test case. Given a set of available test systems and a large number of test cases, the problem is to determine for each test case, which of the test systems are suitable for executing the test case, and to provide the mapping that associates the test case elements (the logical network topology) with the appropriate elements of the test system (the physical network topology). We studied a real industrial environment where this problem was originally handled by a simple software procedure that was very slow in many cases, and also failed to provide thorough coverage of each network's elements. In this paper, we represent both the test systems and the test cases as graphs, and develop a new prototype algorithm that a) determines whether or not a test case can be mapped to a subgraph of the test system, b) rapidly finds mappings that do exist, and c) exercises diverse sets of network nodes when multiple mappings exist for the test case. The prototype has been implemented and applied to over 10,000 combinations of test cases and test systems, and reduced the computation time by a factor of more than 80 from the original procedure. In addition, relative to a meaningful measure of network topology coverage, the mappings achieved an increased level of thoroughness in exercising the elements of each test system.
|
|
| 35. |
- Strandberg, Per Erik, et al.
(författare)
-
Decision Making and Visualizations Based on Test Results
- 2018
-
Ingår i: Empirical Software Engineering and Measurement, 12th International Symposium on ESEM18. - New York, NY, USA : ACM. - 9781450358231
-
Konferensbidrag (refereegranskat)abstract
- Background: Testing is one of the main methods for quality assurance in the development of embedded software, as well as in software engineering in general. Consequently, test results (and how they are reported and visualized) may substantially influence business decisions in software-intensive organizations. Aims: This case study examines the role of test results from automated nightly software testing and the visualizations for decision making they enable at an embedded systems company in Sweden. In particular, we want to identify the use of the visualizations for supporting decisions from three aspects: in daily work, at feature branch merge, and at release time. Method: We conducted an embedded case study with multiple units of analysis by conducting interviews, questionnaires, using archival data and participant observations. Results: Several visualizations and reports built on top of the test results database are utilized in supporting daily work, merging a feature branch to the master and at release time. Some important visualizations are: lists of failing test cases, easy access to log files, and heatmap trend plots. The industrial practitioners perceived the visualizations and reporting as valuable, however they also mentioned several areas of improvement such as better ways of visualizing test coverage in a functional area as well as better navigation between different views. Conclusions: We conclude that visualizations of test results are a vital decision making tool for a variety of roles and tasks in embedded software development, however the visualizations need to be continuously improved to keep their value for its stakeholders.
|
|
| 36. |
- Strandberg, Per Erik, et al.
(författare)
-
Experience Report : Automated System Level Regression Test Prioritization Using Multiple Factors
- 2016
-
Ingår i: 27th International Symposium on Software Reliability Engineering ISSRE'16.
-
Konferensbidrag (refereegranskat)abstract
- We propose a new method of determining an effective ordering of regression test cases, and describe its implementation as an automated tool called SuiteBuilder developed by Westermo Research and Development AB. The tool generates an efficient order to run the cases in an existing test suite by using expected or observed test duration and combining priorities of multiple factors associated with test cases, including previous fault detection success, interval since last executed, and modifications to the code tested. The method and tool were developed to address problems in the traditional process of regression testing, such as lack of time to run a complete regression suite, failure to detect bugs in time, and tests that are repeatedly omitted. The tool has been integrated into the existing nightly test framework for Westermo software that runs on large-scale data communication systems. In experimental evaluation of the tool, we found significant improvement in regression testing results. The re-ordered test suites finish within the available time, the majority of fault-detecting test cases are located in the first third of the suite, no important test case is omitted, and the necessity for manual work on the suites is greatly reduced.
|
|
| 37. |
- Strandberg, Per Erik, et al.
(författare)
-
Information Flow in Software Testing - An Interview Study with Embedded Software Engineering Practitioners
- 2019
-
Ingår i: IEEE Access. - 2169-3536 .- 2169-3536. ; 7, s. 46434-46453
-
Tidskriftsartikel (refereegranskat)abstract
- Background: In order to make informed decisions, software engineering practitioners need information from testing. However, with the trend of increased automation, there is exponential growth and increased distribution of this information. This paper aims at exploring the information flow in software testing in the domain of embedded systems. Method: Data was collected through semi-structured interviews of twelve experienced practitioners with an average work experience of more than fourteen years working at five organizations in the embedded software industry in Sweden. Seventeen hours of audio recordings were transcribed and anonymized into 130 pages of text that was analyzed by means of thematic analysis. Results: The flow of information in software testing can be represented as feedback loops involving stakeholders, software artifacts, test equipment, and test results. The six themes that affect the information flow are how organizations conduct testing and trouble shooting, communication, processes, technology, artifacts, and the organization of the company. Seven main challenges for the flow of information in software testing are comprehending the objectives and details of testing; root cause identification; poor feedback; postponed testing; poor artifacts and traceability; poor tools and test infrastructure; and distances. Finally, five proposed approaches for enhancing the flow are: close collaboration between roles; fast feedback; custom test report automation; test results visualization; and the use of suitable tools and frameworks. Conclusions: The results indicate that there are many opportunities to improve the flow of information in software testing: a first mitigation step is to better understand the challenges and approaches. Future work is needed to realize this in practice, for example, on how to shorten feedback cycles between roles, as well as how to enhance exploration and visualization of test results.
|
|
| 38. |
- Tahvili, Sahar
(författare)
-
A Decision Support System for Integration Test Selection
- 2016
-
Licentiatavhandling (övrigt vetenskapligt/konstnärligt)abstract
- Software testing generally suffers from time and budget limitations. Indiscriminately executing all available test cases leads to sub-optimal exploitation of testing resources. Selecting too few test cases for execution on the other hand might leave a large number of faults undiscovered. Test case selection and prioritization techniques can lead to more efficient usage of testing resources and also early detection of faults. Test case selection addresses the problem of selecting a subset of an existing set of test cases, typically by discarding test cases that do not add any value in improving the quality of the software under test. Test case prioritization schedules test cases for execution in an order to increase their effectiveness at achieving some performance goals such as: earlier fault detection, optimal allocation of testing resources and reducing overall testing effort. In practice, prioritized selection of test cases requires the evaluation of different test case criteria, and therefore, this problem can be formulated as a multi-criteria decision making problem. As the number of decision criteria grows, application of a systematic decision making solution becomes a necessity. In this thesis, we propose a tool-supported framework using a decision support system, for prioritizing and selecting integration test cases in embedded system development. The framework provides a complete loop for selecting the best candidate test case for execution based on a finite set of criteria. The results of multiple case studies, done on a train control management subsystem from Bombardier Transportation AB in Sweden, demonstrate how our approach helps to select test cases in a systematic way. This can lead to early detection of faults while respecting various criteria. Also, we have evaluated a customized return on investment metric to quantify the economic benefits in optimizing system integration testing using our framework.
|
|
| 39. |
- Tahvili, Sahar, et al.
(författare)
-
Cost-Benefit Analysis of Using Dependency Knowledge at Integration Testing
- 2016. - 7
-
Ingår i: Product-Focused Software Process Improvement. - Cham : Springer International Publishing. - 9783319490939 - 9783319490946 ; , s. 268-284, s. 268-284
-
Konferensbidrag (refereegranskat)abstract
- In software system development, testing can take considerable time and resources, and there are numerous examples in the literature of how to improve the testing process. In particular, methods for selection and prioritization of test cases can play a critical role in efficient use of testing resources. This paper focuses on the problem of selection and ordering of integration-level test cases. Integration testing is performed to evaluate the correctness of several units in composition. Further, for reasons of both effectiveness and safety, many embedded systems are still tested manually. To this end, we propose a process, supported by an online decision support system, for ordering and selection of test cases based on the test result of previously executed test cases. To analyze the economic efficiency of such a system, a customized return on investment (ROI) metric tailored for system integration testing is introduced. Using data collected from the development process of a large-scale safety-critical embedded system, we perform Monte Carlo simulations to evaluate the expected ROI of three variants of the proposed new process. The results show that our proposed decision support system is beneficial in terms of ROI at system integration testing and thus qualifies as an important element in improving the integration testing process.
|
|
| 40. |
- Tahvili, Sahar, et al.
(författare)
-
Dynamic Integration Test Selection Based on Test Case Dependencies
- 2016
-
Ingår i: 2016 IEEE NINTH INTERNATIONAL CONFERENCE ON SOFTWARE TESTING, VERIFICATION AND VALIDATION WORKSHOPS (ICSTW). - Chicago, United States. - 9781509036745 ; , s. 277-286
-
Konferensbidrag (refereegranskat)abstract
- Prioritization, selection and minimization of test cases are well-known problems in software testing. Test case prioritization deals with the problem of ordering an existing set of test cases, typically with respect to the estimated likelihood of detecting faults. Test case selection addresses the problem of selecting a subset of an existing set of test cases, typically by discarding test cases that do not add any value in improving the quality of the software under test. Most existing approaches for test case prioritization and selection suffer from one or several drawbacks. For example, they to a large extent utilize static analysis of code for that purpose, making them unfit for higher levels of testing such as integration testing. Moreover, they do not exploit the possibility of dynamically changing the prioritization or selection of test cases based on the execution results of prior test cases. Such dynamic analysis allows for discarding test cases that do not need to be executed and are thus redundant. This paper proposes a generic method for prioritization and selection of test cases in integration testing that addresses the above issues. We also present the results of an industrial case study where initial evidence suggests the potential usefulness of our approach in testing a safety-critical train control management subsystem.
|
|
| 41. |
- Tahvili, Sahar, et al.
(författare)
-
Towards Earlier Fault Detection by Value-Driven Prioritization of Test Cases Using Fuzzy TOPSIS
- 2016. - 9
-
Ingår i: Information Technology: New Generations. - Cham : Springer International Publishing. - 9783319324661 - 9783319324678 ; , s. 745-759
-
Konferensbidrag (refereegranskat)abstract
- In industrial software testing, development projects typically set up and maintain test suites containing large numbers of test cases. Executing a large number of test cases can be expensive in terms of effort and wall-clock time. Moreover, indiscriminate execution of all available test cases typically lead to sub-optimal use of testing resources. On the other hand, selecting too few test cases for execution might leave a large number of faults undiscovered. Limiting factors such as allocated budget and time constraints for testing further emphasizes the importance of test case prioritization in order to identify test cases that enable earlier detection of faults while respecting such constraints. In this paper, we propose a multi-criteria decision making approach for prioritizing test cases in order to detect faults earlier. This is achieved by applying the TOPSIS (Technique for Order of Preference by Similarity to Ideal Solution) decision making technique combined with fuzzy principles. Our solution is based on important criteria such as fault detection probability, execution time, complexity, and other test case properties. By applying the approach on a train control management subsystem from Bombardier Transportation in Sweden, we demonstrate how it helps, in a systematic way, to identify test cases that can lead to early detection of faults while respecting various criteria.
|
|
| 42. |
- Wiklund, Kristian, 1972-
(författare)
-
Impediments for Automated Software Test Execution
- 2015
-
Doktorsavhandling (övrigt vetenskapligt/konstnärligt)abstract
- Automated software test execution is a critical part of the modern software development process, where rapid feedback on the product quality is expected. It is of high importance that impediments related to test execution automation are prevented and removed as quickly as possible. An enabling factor for all types of improvement is to understand the nature of what is to be improved. The goal with this thesis is to further the knowledge about common problems encountered by software developers using test execution automation, in order to enable improvement of test execution automation in industrial software development. The research has been performed through industrial case studies and literature reviews. The analysis of the data have primarily been performed using qualitative methods, searching for patterns, themes, and concepts in the data. Our main findings include: (a) a comprehensive list of potential impediments reported in the published body of knowledge on test execution automation, (b) an in-depth analysis of how such impediments may affect the performance of a development team, and (c) a proposal for a qualitative model of interactions between the main groups of phenomena that contribute to the formation of impediments in a test execution automation project. In addition to this, we contribute qualitative and quantitative empirical data from our industrial case studies. Through our results, we find that test execution automation is a commonly under-estimated activity, not only in terms of resources but also in terms of the complexity of the work. There is a clear tendency to perform the work ad hoc, down-prioritize the automation in favor of other activities, and ignore the long-term effects in favor of short-term gains. This is both a technical and a cultural problem that need to be managed by awareness of the problems that may arise, and also have to be solved in the long term through education and information. We conclude by proposing a theoretical model of the socio-technical system that needs to be managed to be successful with test execution automation.
|
|
| 43. |
- Wiklund, K., et al.
(författare)
-
Impediments for software test automation : A systematic literature review
- 2017
-
Ingår i: Software testing, verification & reliability. - : John Wiley and Sons Ltd. - 0960-0833 .- 1099-1689. ; 27:8
-
Tidskriftsartikel (refereegranskat)abstract
- Automated software testing is a critical enabler for modern software development, where rapid feedback on the product quality is expected. To make the testing work well, it is of high importance that impediments related to test automation are prevented and removed quickly. An enabling factor for all types of improvement is to understand the nature of what is to be improved. We have performed a systematic literature review of reported impediments related to software test automation to contribute to this understanding. In this paper, we present the results from the systematic literature review: The list of identified publications, a categorization of identified impediments, and a qualitative discussion of the impediments proposing a socio-technical system model of the use and implementation of test automation.
|
|
