| 1. |
- Brahneborg, Daniel, 1970-, et al.
(author)
-
Round-Trip Time Anomaly Detection
- 2018
-
In: ICPE '18 Proceedings of the 2018 ACM/SPEC International Conference on Performance Engineering. - New York, NY, USA : ACM. - 9781450350952 ; , s. 107-114
-
Conference paper (peer-reviewed)abstract
- Mobile text messages (SMS) are sometimes used for authentication, which requires short and reliable delivery times. The observed round-trip times when sending an SMS message provide valuable information on the quality of the connection. In this industry paper, we propose a method for detecting round-trip time anomalies, where the exact distribution is unknown, the variance is several orders of magnitude, and there are lots of shorter spikes that should be ignored. In particular, we show that using an adaption of Double Seasonal Exponential Smoothing to reduce the content dependent variations, followed by the Remedian to find short-term and long-term medians, successfully identifies larger groups of outliers. As training data for our method we use log files from a live SMS gateway. In order to verify the effectiveness of our approach, we utilize simulated data. Our contributions are a description on how to isolate content dependent variations, and the sequence of steps to find significant anomalies in big data.
|
|
| 2. |
- Flemström, Daniel, et al.
(author)
-
A Research Roadmap for Test Design in Automated Integration Testing of Vehicular Systems
- 2016. - 9
-
In: FASSI 2016. - : International Academy, Research and Industry Association (IARIA). - 9781612084978 ; , s. 18-23
-
Conference paper (peer-reviewed)abstract
- An increasing share of the innovations emerging in the vehicular industry are implemented in software. Consequently, vehicular electrical systems are becoming more and more complex with an increasing number of functions, computational nodes and complex sensors, e.g., cameras and radars. The introduction of autonomous functional components, such as advanced driver assistance systems, highlight the foreseeable complexity of different parts of the system interacting with each other and with the human driver. It is of utmost importance that the testing effort can scale with this increasing complexity. In this paper, we review the challenges that we are facing in integration testing of complex embedded vehicular systems. Further, based on these challenges we outline a set of research directions for semi-automated or automated test design and execution in integration testing of vehicular systems. While the discussion is exemplified with our hands-on experience of the automotive industry, much of the concepts can be generalised to a broader setting of complex embedded systems.
|
|
| 3. |
- Flemström, Daniel, et al.
(author)
-
Exploring Test Overlap in System Integration : An Industrial Case Study
- 2016
-
In: 42nd Euromicro Conference series on Software Engineering and Advanced Applications SEAA 2016. - 9781509028191 ; , s. 303-312
-
Conference paper (peer-reviewed)abstract
- Tougher safety regulations, global competition and ever increasing complexity of embedded software puts extensive pressure on the effectiveness of the software testing procedures. Previous studies have found that there exist overlaps (i.e., multiple instances of highly similar test cases) and even redundancies in the software testing process. Such overlap has been found between versions, variants and integration levels, but primarily at unit test level. Given large embedded systems involving many subsystems, does overlap exist within the system integration testing as well? In this paper, we present an industrial case study, aiming to a) evaluate if there exist test overlaps within the given context, b) if so, investigate how these overlaps are distributed, and c) find ways of reducing test effort by investigating how the knowledge of overlaps and their distribution may be used for finding candidate test cases for automation, maintenance or even removal. We have studied manual test cases, written in natural language, at a large vehicular manufacturer in Sweden. In particular, we have collected and analyzed test cases from the system integration testing levels of four different projects of a vehicle control management system. Using a similarity function, we evaluate if any overlaps between test cases exist, and where. We found that overlaps do exist within the system integration level, particularly in the form of partial test step sequences. However, very few test cases overlapped in their entirety. Some candidates for test step automation and update propagation were identified, but none for easy removal.
|
|
| 4. |
- Flemström, Daniel, et al.
(author)
-
From natural language requirements to passive test cases using guarded assertions
- 2018
-
In: Proceedings - 2018 IEEE 18th International Conference on Software Quality, Reliability, and Security, QRS 2018. - : Institute of Electrical and Electronics Engineers Inc.. - 9781538677575 ; , s. 470-481
-
Conference paper (peer-reviewed)abstract
- In large-scale embedded system development, requirements are often expressed in natural language. Translating these requirements to executable test cases, while keeping the test cases and requirements aligned, is a challenging task. While such a transformation typically requires extensive domain knowledge, we show that a systematic process in combination with passive testing would facilitate the translation as well as linking the requirements to tests. Passive testing approaches observe the behavior of the system and test their correctness without interfering with the normal behavior. We use a specific approach to passive testing: guarded assertions (G/A). This paper presents a method for transforming system requirements expressed in natural language into G/As. We further present a proof of concept evaluation, performed at Bombardier Transportation Sweden AB, in which we show how the process would be used, together with practical advice of the reasoning behind the translation steps.
|
|
| 5. |
- Flemström, Daniel, 1971-
(author)
-
Industrial System Level Test Automation
- 2021
-
Doctoral thesis (other academic/artistic)abstract
- Vehicular software systems control and monitor many safety-critical functions, such as automated emergency brakes and anti-spin. These functions are integrated and tested at system level to ensure that the entire system works as intended.Traditionally, these functions or selected combinations of functions are tested in isolation. Although desired, rigorous testing of combinations of functions prior to deployment is seldom possible. One reason is the overwhelming work required to write new test cases for the nearly infinite combinations of functions and driving scenarios. Since software testing already accounts for up to 60% of the software development cost and the execution time in test rigs is expensive, further testing must be achieved by running test cases more in parallel. Moreover, new test cases must be reusable in different driving scenarios to cover more combinations of functions. One solution is to express the test logic in a new way so that it can be executed in parallel, independently of other tests and independent of the input stimuli. This would allow reusing the test logic for different sequences or drive scenarios. Passive testing is one such approach that has not yet been used much for vehicular software due to perceived difficulties, although it is well-established in other domains. One particular problem is how to specify and execute passive test cases for vehicular systems in an, for the test engineer, intuitive and straightforward manner. Further, there is a lack of tool support and knowledge on efficiently applying passive testing in an industrial context. Thus, the overall research goal of this thesis is to propose and evaluate industrially applicable methods and tools for passive testing at the system level of vehicular software systems. The research is based on a series of papers and case studies within the vehicular industry. In contrast to existing specification languages based on formal mathematical expressions, considerable effort was spent creating an intuitive language and an interactive tool, simple but powerful enough, to encourage the industrial application of passive testing. The main contributions include an easy-to-write and easy-to-read description language for passive test cases, an interactive development environment, and knowledge on how to succeed in passive testing in an industrial software development process. The findings of this thesis contribute to making passive testing a viable method for system-level testing and a way of reusing test logic. In the case of a studied train control management system, using passive testing may reuse up to 50% of the test logic for the safety-related requirements and 10% of the non-safety-related requirements.
|
|
| 6. |
- Flemström, Daniel, et al.
(author)
-
Similarity-based prioritization of test case automation
- 2018
-
In: Software quality journal. - : Springer Science and Business Media LLC. - 0963-9314 .- 1573-1367. ; 26:4, s. 1421-1449
-
Journal article (peer-reviewed)abstract
- The importance of efficient software testing procedures is driven by an ever increasing system complexity as well as global competition. In the particular case of manual test cases at the system integration level, where thousands of test cases may be executed before release, time must be well spent in order to test the system as completely and as efficiently as possible. Automating a subset of the manual test cases, i.e, translating the manual instructions to automatically executable code, is one way of decreasing the test effort. It is further common that test cases exhibit similarities, which can be exploited through reuse when automating a test suite. In this paper, we investigate the potential for reducing test effort by ordering the test cases before such automation, given that we can reuse already automated parts of test cases. In our analysis, we investigate several approaches for prioritization in a case study at a large Swedish vehicular manufacturer. The study analyzes the effects with respect to test effort, on four projects with a total of 3919 integration test cases constituting 35,180 test steps, written in natural language. The results show that for the four projects considered, the difference in expected manual effort between the best and the worst order found is on average 12 percentage points. The results also show that our proposed prioritization method is nearly as good as more resource demanding meta-heuristic approaches at a fraction of the computational time. Based on our results, we conclude that the order of automation is important when the set of test cases contain similar steps (instructions) that cannot be removed, but are possible to reuse. More precisely, the order is important with respect to how quickly the manual test execution effort decreases for a set of test cases that are being automated.
|
|
| 7. |
- Flemström, Daniel, 1971-
(author)
-
Similarity-Based Test Effort Reduction
- 2017
-
Licentiate thesis (other academic/artistic)abstract
- Embedded computer systems are all around us. We find them in everything, from dishwashers to cars and airplanes. They must always work correctly and moreover, often within certain time constraints. The software of such a system can be very large and complex, e.g. in the case of a car or a train. Hence, we develop the software for embedded systems in smaller, manageable, parts. These parts can be successively integrated, until they form the complete software for the embedded system, possibly at different levels. This phase of the development process is called the system integration phase and is one of the most critical phases in the development of embedded systems. In this phase, substantial effort is spent on testing activities.Studies have found that a considerable amount of test effort is wasteful due to people, unknowingly or by necessity, performing similar (or even overlapping) test activities. Consequently, test cases may end up to be similar, partially or wholly. We identified such test similarities in a case study of 2500 test cases, written in natural language, from four different projects in the embedded vehicular domain. Such information can be used for reducing effort when maintaining or automating similar test cases.In another case study in the same domain, we investigated several approaches for prioritizing test cases to automate with the objective to reduce manual test effort as quick as possible given that similar automated tests could be reused (similarity-based reuse). We analyzed how the automation order affects the test effort for four projects with a total of 3919 integration test cases, written in natural language. The results showed that similarity-based reuse of automated test case script code, and the best-performing automation order can reduce the expected manual test effort with 20 percentage points.Another way of reducing test effort is to reuse test artifacts from one level of integration to another, instead of duplicating them. We studied such reuse methods, that we denote vertical reuse, in a systematic mapping study. While the results from of our systematic mapping study showed the viability of vertical test reuse methods, our industrial case studies showed that keeping track of similarities and test overlaps is both possible and feasible for test effort reduction. We further conclude that the test case automation order affects the manual test execution effort when there exist similar steps that cannot be removed, but are possible to reuse with respect to test script code.
|
|
| 8. |
- Flemström, Daniel, et al.
(author)
-
Vertical Test Reuse for Embedded Systems : A Systematic Mapping Study
- 2015. - 11
-
In: 2015 41st Euromicro Conference on Software Engineering and Advanced Applications. - : Conference Publishing Services. - 9781467375856 ; , s. 317-324
-
Conference paper (peer-reviewed)abstract
- Vertical test reuse refers to the the reuse of test cases or other test artifacts over different integration levels in the software or system engineering process. Vertical test reuse has previously been proposed for reducing test effort and improving test effectiveness, particularly for embedded system development. The goal of this study is to provide an overview of the state of the art in the field of vertical test reuse for embedded system development. For this purpose, a systematic mapping study has been performed, identifying 11 papers on vertical test reuse for embedded systems. The primary result from the mapping is a classification of published work on vertical test reuse in the embedded system domain, covering motivations for reuse, reuse techniques, test levels and reusable test artifacts considered, and to what extent the effects of reuse have been evaluated.
|
|
| 9. |
|
|
| 10. |
- Abbas, Muhammad, et al.
(author)
-
Automated Reuse Recommendation of Product Line Assets Based on Natural Language Requirements
- 2020
-
In: Lecture Notes in Computer Science. - Cham : Springer Science and Business Media Deutschland GmbH. - 9783030646936 ; , s. 173-189, s. 173-189, s. 173-189
-
Conference paper (peer-reviewed)abstract
- Software product lines (SPLs) are based on reuse rationale to aid quick and quality delivery of complex products at scale. Deriving a new product from a product line requires reuse analysis to avoid redundancy and support a high degree of assets reuse. In this paper, we propose and evaluate automated support for recommending SPL assets that can be reused to realize new customer requirements. Using the existing customer requirements as input, the approach applies natural language processing and clustering to generate reuse recommendations for unseen customer requirements in new projects. The approach is evaluated both quantitatively and qualitatively in the railway industry. Results show that our approach can recommend reuse with 74% accuracy and 57.4% exact match. The evaluation further indicates that the recommendations are relevant to engineers and can support the product derivation and feasibility analysis phase of the projects. The results encourage further study on automated reuse analysis on other levels of abstractions.
|
|
| 11. |
|
|
| 12. |
- Abbas, Muhammad, et al.
(author)
-
Keywords-based test categorization for Extra-Functional Properties
- 2020
-
In: 2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). - : IEEE. - 9781728110752 ; , s. 153-156
-
Conference paper (peer-reviewed)abstract
- Categorizing existing test specifications can provide insights on coverage of the test suite to extra-functional properties. Manual approaches for test categorization can be time-consuming and prone to error. In this short paper, we propose a semi-automated approach for semantic keywords-based textual test categorization for extra-functional properties. The approach is the first step towards coverage-based test case selection based on extra-functional properties. We report a preliminary evaluation of industrial data for test categorization for safety aspects. Results show that keyword-based approaches can be used to categorize tests for extra-functional properties and can be improved by considering contextual information of keywords.
|
|
| 13. |
- Abbas, Muhammad, et al.
(author)
-
MBRP : Model-Based Requirements Prioritization Using PageRank Algorithm
- 2019
-
In: 2019 26th Asia-Pacific Software Engineering Conference (APSEC). - : IEEE conference proceedings. - 9781728146485 ; , s. 31-38
-
Conference paper (peer-reviewed)abstract
- Requirements prioritization plays an important role in driving project success during software development. Literature reveals that existing requirements prioritization approaches ignore vital factors such as interdependency between requirements. Existing requirements prioritization approaches are also generally time-consuming and involve substantial manual effort. Besides, these approaches show substantial limitations in terms of the number of requirements under consideration. There is some evidence suggesting that models could have a useful role in the analysis of requirements interdependency and their visualization, contributing towards the improvement of the overall requirements prioritization process. However, to date, just a handful of studies are focused on model-based strategies for requirements prioritization, considering only conflict-free functional requirements. This paper uses a meta-model-based approach to help the requirements analyst to model the requirements, stakeholders, and inter-dependencies between requirements. The model instance is then processed by our modified PageRank algorithm to prioritize the given requirements. An experiment was conducted, comparing our modified PageRank algorithm's efficiency and accuracy with five existing requirements prioritization methods. Besides, we also compared our results with a baseline prioritized list of 104 requirements prepared by 28 graduate students. Our results show that our modified PageRank algorithm was able to prioritize the requirements more effectively and efficiently than the other prioritization methods.
|
|
| 14. |
- Abbas, Muhammad, et al.
(author)
-
On the relationship between similar requirements and similar software : A case study in the railway domain
- 2023
-
In: Requirements Engineering. - : Springer Science and Business Media Deutschland GmbH. - 0947-3602 .- 1432-010X. ; 28, s. 23-47
-
Journal article (peer-reviewed)abstract
- Recommender systems for requirements are typically built on the assumption that similar requirements can be used as proxies to retrieve similar software. When a stakeholder proposes a new requirement, natural language processing (NLP)-based similarity metrics can be exploited to retrieve existing requirements, and in turn, identify previously developed code. Several NLP approaches for similarity computation between requirements are available. However, there is little empirical evidence on their effectiveness for code retrieval. This study compares different NLP approaches, from lexical ones to semantic, deep-learning techniques, and correlates the similarity among requirements with the similarity of their associated software. The evaluation is conducted on real-world requirements from two industrial projects from a railway company. Specifically, the most similar pairs of requirements across two industrial projects are automatically identified using six language models. Then, the trace links between requirements and software are used to identify the software pairs associated with each requirements pair. The software similarity between pairs is then automatically computed with JPLag. Finally, the correlation between requirements similarity and software similarity is evaluated to see which language model shows the highest correlation and is thus more appropriate for code retrieval. In addition, we perform a focus group with members of the company to collect qualitative data. Results show a moderately positive correlation between requirements similarity and software similarity, with the pre-trained deep learning-based BERT language model with preprocessing outperforming the other models. Practitioners confirm that requirements similarity is generally regarded as a proxy for software similarity. However, they also highlight that additional aspect comes into play when deciding software reuse, e.g., domain/project knowledge, information coming from test cases, and trace links. Our work is among the first ones to explore the relationship between requirements and software similarity from a quantitative and qualitative standpoint. This can be useful not only in recommender systems but also in other requirements engineering tasks in which similarity computation is relevant, such as tracing and change impact analysis.
|
|
| 15. |
- Abbas, Muhammad, et al.
(author)
-
Product line adoption in industry : an experience report from the railway domain
- 2020
-
In: SPLC '20: Proceedings of the 24th ACM Conference on Systems and Software Product Line: Volume A. - New York, NY, USA : Association for Computing Machinery. - 9781450375696 ; , s. 130-141, s. 130-141
-
Conference paper (peer-reviewed)abstract
- The software system controlling a train is typically deployed on various hardware architectures and must process various signals across those deployments. The increase of such customization scenarios and the needed adherence of the software to various safety standards in different application domains has led to the adoption of product line engineering within the railway domain. This paper explores the current state-of-practice of software product line development within a team developing industrial embedded software for a train propulsion control system. Evidence is collected using a focus group session with several engineers and through inspection of archival data. We report several benefits and challenges experienced during product line adoption and deployment. Furthermore, we identify and discuss improvement opportunities, focusing mainly on product line evolution and test automation.
|
|
| 16. |
- Abbas, Muhammad
(author)
-
Requirements-Level Reuse Recommendation and Prioritization of Product Line Assets
- 2021
-
Licentiate thesis (other academic/artistic)abstract
- Software systems often target a variety of different market segments. Targeting varying customer requirements requires a product-focused development process. Software Product Line (SPL) engineering is one possible approach based on reuse rationale to aid quick delivery of quality product variants at scale. SPLs reuse common features across derived products while still providing varying configuration options. The common features, in most cases, are realized by reusable assets. In practice, the assets are reused in a clone-and-own manner to reduce the upfront cost of systematic reuse. Besides, the assets are implemented in increments, and requirements prioritization also has to be done. In this context, the manual reuse analysis and prioritization process become impractical when the number of derived products grows. Besides, the manual reuse analysis process is time-consuming and heavily dependent on the experience of engineers.In this licentiate thesis, we study requirements-level reuse recommendation and prioritization for SPL assets in industrial settings. We first identify challenges and opportunities in SPLs where reuse is done in a clone-and-own manner. We then focus on one of the identified challenges: requirements-based SPL assets reuse and provide automated support for identifying reuse opportunities for SPL assets based on requirements. Finally, we provide automated support for requirements prioritization in the presence of dependencies resulting from reuse.
|
|
| 17. |
- Abbaspour Asadollah, Sara, et al.
(author)
-
10 Years of research on debugging concurrent and multicore software : a systematic mapping study
- 2017
-
In: Software quality journal. - : Springer Science and Business Media LLC. - 0963-9314 .- 1573-1367. ; 25:1, s. 49-82
-
Journal article (peer-reviewed)abstract
- Debugging – the process of identifying, localizing and fixing bugs – is a key activity in software development. Due to issues such as non-determinism and difficulties of reproducing failures, debugging concurrent software is significantly more challenging than debugging sequential software. A number of methods, models and tools for debugging concurrent and multicore software have been proposed, but the body of work partially lacks a common terminology and a more recent view of the problems to solve. This suggests the need for a classification, and an up-to-date comprehensive overview of the area. This paper presents the results of a systematic mapping study in the field of debugging of concurrent and multicore software in the last decade (2005– 2014). The study is guided by two objectives: (1) to summarize the recent publication trends and (2) to clarify current research gaps in the field.Through a multi-stage selection process, we identified 145 relevant papers. Based on these, we summarize the publication trend in the field by showing distribution of publications with respect to year , publication venues , representation of academia and industry , and active research institutes . We also identify research gaps in the field based on attributes such as types of concurrency bugs, types of debugging processes , types of research and research contributions.The main observations from the study are that during the years 2005–2014: (1) there is no focal conference or venue to publish papers in this area, hence a large variety of conferences and journal venues (90) are used to publish relevant papers in this area; (2) in terms of publication contribution, academia was more active in this area than industry; (3) most publications in the field address the data race bug; (4) bug identification is the most common stage of debugging addressed by articles in the period; (5) there are six types of research approaches found, with solution proposals being the most common one; and (6) the published papers essentially focus on four different types of contributions, with ”methods” being the type most common one.We can further conclude that there is still quite a number of aspects that are not sufficiently covered in the field, most notably including (1) exploring correction and fixing bugs in terms of debugging process; (2) order violation, suspension and starvation in terms of concurrency bugs; (3) validation and evaluation research in the matter of research type; (4) metric in terms of research contribution. It is clear that the concurrent, parallel and multicore software community needs broader studies in debugging.This systematic mapping study can help direct such efforts.
|
|
| 18. |
- Abbaspour Asadollah, Sara, et al.
(author)
-
A Model for Systematic Monitoring and Debugging of Starvation Bugs in Multicore Software
- 2016
-
In: Proceedings of the 1st International Workshop on Specification, Comprehension, Testing, and Debugging of Concurrent Programs (SCTDCP 2016). - New York, NY, USA : ACM. - 9781450345101 ; , s. 7-11
-
Conference paper (peer-reviewed)abstract
- With the development of multicore hardware, concurrent, parallel and multicore software are becoming increasingly popular. Software companies are spending a huge amount of time and resources to nd and debug the bugs. Among all types of software bugs, concurrency bugs are also important and troublesome. This type of bugs is increasingly becoming an issue particularly due to the growing prevalence of multicore hardware. In this position paper, we propose a model for monitoring and debugging Starvation bugs as a type of concurrency bugs in multicore software. The model is composed into three phases: monitoring, detecting and debugging. The monitoring phase can support detecting phase by storing collected data from the system execution. The detecting phase can support debugging phase by comparing the stored data with starvation bug's properties, and the debugging phase can help in reproducing and removing the Starvation bug from multicore software. Our intention is that our model is the basis for developing tool(s) to enable solving Starvation bugs in software for multicore platforms.
|
|
| 19. |
|
|
| 20. |
- Abbaspour Asadollah, Sara, et al.
(author)
-
A Runtime Verification Tool for Detecting Concurrency Bugs in FreeRTOS Embedded Software
- 2018
-
In: Proceedings - 17th International Symposium on Parallel and Distributed Computing, ISPDC 2018. - : Institute of Electrical and Electronics Engineers Inc.. - 9781538653302 ; , s. 172-179
-
Conference paper (peer-reviewed)abstract
- This article presents a runtime verification tool for embedded software executing under the open source real-time operating system FreeRTOS. The tool detects and diagnoses concurrency bugs such as deadlock, starvation, and suspension based-locking. The tool finds concurrency bugs at runtime without debugging and tracing the source code. The tool uses the Tracealyzer tool for logging relevant events. Analysing the logs, our tool can detect the concurrency bugs by applying algorithms for diagnosing each concurrency bug type individually. In this paper, we present the implementation of the tool, as well as its functional architecture, together with illustration of its use. The tool can be used during program testing to gain interesting information about embedded software executions. We present initial results of running the tool on some classical bug examples running on an AVR 32-bit board SAM4S.
|
|
| 21. |
- Abbaspour Asadollah, Sara, et al.
(author)
-
A Study on Concurrency Bugs in an Open Source Software
- 2016
-
In: IFIP Advances in Information and Communication Technology, vol. 472. - Cham : Springer International Publishing. - 9783319392240 ; , s. 16-31
-
Conference paper (peer-reviewed)abstract
- Concurrent programming puts demands on software debugging and testing, as concurrent software may exhibit problems not present in sequential software, e.g., deadlocks and race conditions. In aiming to increase efficiency and effectiveness of debugging and bug-fixing for concurrent software, a deep understanding of concurrency bugs, their frequency and fixingtimes would be helpful. Similarly, to design effective tools and techniques for testing and debugging concurrent software understanding the differences between non-concurrency and concurrency bugs in real-word software would be useful.
|
|
| 22. |
|
|
| 23. |
- Abbaspour Asadollah, Sara, et al.
(author)
-
Concurrency bugs in open source software : a case study
- 2017
-
In: Journal of Internet Services and Applications. - : Springer London. - 1867-4828 .- 1869-0238. ; 8:1
-
Journal article (peer-reviewed)abstract
- Concurrent programming puts demands on software debugging and testing, as concurrent software may exhibit problems not present in sequential software, e.g., deadlocks and race conditions. In aiming to increase efficiency and effectiveness of debugging and bug-fixing for concurrent software, a deep understanding of concurrency bugs, their frequency and fixing-times would be helpful. Similarly, to design effective tools and techniques for testing and debugging concurrent software, understanding the differences between non-concurrency and concurrency bugs in real-word software would be useful. This paper presents an empirical study focusing on understanding the differences and similarities between concurrency bugs and other bugs, as well as the differences among various concurrency bug types in terms of their severity and their fixing time, and reproducibility. Our basis is a comprehensive analysis of bug reports covering several generations of five open source software projects. The analysis involves a total of 11860 bug reports from the last decade, including 351 reports related to concurrency bugs. We found that concurrency bugs are different from other bugs in terms of their fixing time and severity while they are similar in terms of reproducibility. Our findings shed light on concurrency bugs and could thereby influence future design and development of concurrent software, their debugging and testing, as well as related tools.
|
|
| 24. |
- Abbaspour Asadollah, Sara, et al.
(author)
-
Runtime Verification for Detecting Suspension Bugs in Multicore and Parallel Software
- 2017
-
In: Proceedings - 10th IEEE International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2017. - 9781509066766 ; , s. 77-80
-
Conference paper (peer-reviewed)abstract
- Multicore hardware development increases the popularity of parallel and multicore software, while testing and debugging the software become more difficult, frustrating and costly. Among all types of software bugs, concurrency bugs are both important and troublesome. This type of bugs is increasingly becoming an issue, particularly due to the growing prevalence of multicore hardware. Suspension-based-locking bug is one type of concurrency bugs. This position paper proposes a model based on runtime verification and reflection technique in the context of multicore and parallel software to monitor and detect suspension-based-locking bugs. The model is not only able to detect faults, but also diagnose and even repair them. The model is composed of four layers: Logging, Monitoring, Suspension Bug Diagnosis and Mitigation. The logging layer will observe the events and save them into a file system. The monitoring layer will detect the presents of bugs in the software. The suspension bug diagnosis will identify Suspension bugs by comparing the captured data with the suspension bug properties. Finally, the mitigation layer will reconfigure the software to mitigate the suspension bugs. A functional architecture of a runtime verification tool is also proposed in this paper. This architecture is based on the proposed model and is comprised of different modules.
|
|
| 25. |
- Abbaspour Asadollah, Sara, et al.
(author)
-
Towards Classification of Concurrency Bugs Based on Observable Properties
- 2015
-
In: Proceedings - 1st International Workshop on Complex Faults and Failures in Large Software Systems, COUFLESS 2015. - 9781479919345 ; , s. 41-47
-
Conference paper (peer-reviewed)abstract
- In software engineering, classification is a way to find an organized structure of knowledge about objects. Classification serves to investigate the relationship between the items to be classified, and can be used to identify the current gaps in the field. In many cases users are able to order and relate objects by fitting them in a category. This paper presents initial work on a taxonomy for classification of errors (bugs) related to concurrent execution of application level software threads. By classifying concurrency bugs based on their corresponding observable properties, this research aims to examine and structure the state of the art in this field, as well as to provide practitioner support for testing and debugging of concurrent software. We also show how the proposed classification, and the different classes of bugs, relates to the state of the art in the field by providing a mapping of the classification to a number of recently published papers in the software engineering field.
|
|
| 26. |
- Bilic, Damir, et al.
(author)
-
Detecting inconsistencies in annotated product line models
- 2020
-
In: SPLC '20: Proceedings of the 24th ACM Conference on Systems and Software Product Line: Volume A. - New York, NY, USA : Association for Computing Machinery. - 9781450375696 ; , s. 252-262
-
Conference paper (peer-reviewed)abstract
- Model-based product line engineering applies the reuse practices from product line engineering with graphical modeling for the specification of software intensive systems. Variability is usually described in separate variability models, while the implementation of the variable systems is specified in system models that use modeling languages such as SysML. Most of the SysML modeling tools with variability support, implement the annotation-based modeling approach. Annotated product line models tend to be error-prone since the modeler implicitly describes every possible variant in a single system model. To identifying variability-related inconsistencies, in this paper, we firstly define restrictions on the use of SysML for annotative modeling in order to avoid situations where resulting instances of the annotated model may contain ambiguous model constructs. Secondly, inter-feature constraints are extracted from the annotated model, based on relations between elements that are annotated with features. By analyzing the constraints, we can identify if the combined variability- and system model can result in incorrect or ambiguous instances. The evaluation of our prototype implementation shows the potential of our approach by identifying inconsistencies in the product line model of our industrial partner which went undetected through several iterations of the model.
|
|
| 27. |
- Bilic, Damir
(author)
-
Managing Variability in SysML Models of Automotive Systems
- 2020
-
Licentiate thesis (other academic/artistic)abstract
- Organizations developing software-intensive systems inevitably face increasing complexity of developed products, mainly due to rapid advancements in all domains of technology. Many such organizations are considering model-based systems engineering (MBSE) practices to cope with the increasing complexity. The use of models, as a central role during product design, promises to provide benefits such as enhanced communication among system stakeholders, continuous verification, improved design integrity, traceability between requirements and system artifacts and many more. Additionally, products are often built in many variants. That is especially obvious in the automotive domain, where customers have the ability to configure vehicles with hundreds of configuration options. To deal with the variability, a product line engineering approach is often used. It allows the development of a family of similar software-intensive systems that share a common base while being adapted to individual customer requirements.In this thesis, the overall goal is to evaluate and facilitate the combination of the two mentioned approaches, model-based systems engineering and product line engineering, in an industrial environment. To achieve the main thesis goal, it was divided into three separate research goals.The first goal was to identify challenges when applying an annotation-based approach for variant management in SysML models on a use case provided by Volvo Construction Equipment. The aim was to identify and understand challenges when using existing tool support to manage variants in implementation artifacts of existing products. The second research goal was to identify reuse-related challenges in the ``clone-and-own'' based development process of Volvo CE. Moreover, we assess the effects of model-based product line engineering on the identified challenges. Lastly, the third research goal was to develop an approach for consistency checking between variability- and SysML system models. To achieve that, we develop an integrated tool chain for model-based product line engineering that allows the integration of variable artifacts, which are not documented in system models, into the development process. Secondly, we define and develop an approach for consistency checking between variability models that describe the system in terms of features and implementation models that describe how variability is implemented in the product itself, since such support does not exist in current state of the art tools.In conclusion, based on the results from the results of case studies at Volvo CE, it was shown that model-based product line engineering has the potential to improve communication and highlight implications of variability to stakeholders (e.g. to non-technical staff), improve traceability between variability in requirements and variability in design and implementation, improve consistency through constraints between variants and automate repetitive activities.In other words, it shows potential for improving product quality while reducing the development lead time. However, the evaluation and measurement of improvement will be left for future work as measuring the product quality and lead time requires an organizational roll out of model-based product-line engineering.
|
|
| 28. |
- Bilic, Damir, et al.
(author)
-
Model-Based Product Line Engineering in an Industrial Automotive Context: An Exploratory Case Study
- 2018
-
In: 1st Intl. Workshop on Variability and Evolution of Software-intensive Systems VariVolution'18. - New York, NY, USA : ACM. ; , s. 56-63
-
Conference paper (peer-reviewed)abstract
- Product Line Engineering is an approach to reuse assets of complex systems by taking advantage of commonalities between product families. Reuse within complex systems usually means reuse of artifacts from different engineering domains such as mechanical, electronics and software engineering. Model-based systems engineering is becoming a standard for systems engineering and collaboration within different domains. This paper presents an exploratory case study on initial efforts of adopting Product Line Engineering practices within the model-based systems engineering process at Volvo Construction Equipment (Volvo CE), Sweden. We have used SysML to create overloaded models of the engine systems at Volvo CE. The variability within the engine systems was captured by using the Orthogonal Variability Modeling language. The case study has shown us that overloaded SysML models tend to become complex even on small scale systems, which in turn makes scalability of the approach a major challenge. For successful reuse and to, possibly, tackle scalability, it is necessary to have a database of reusable assets from which product variants can be derived.
|
|
| 29. |
- Bilic, Damir, et al.
(author)
-
Towards a Model-Driven Product Line Engineering Process : An Industrial Case Study
- 2020
-
In: Proceedings of the 13th Innovations in Software Engineering Conference on Formerly Known as India Software Engineering Conference. - New York, NY, USA : Association for Computing Machinery.
-
Conference paper (peer-reviewed)abstract
- Many organizations developing software-intensive systems face challenges with high product complexity and large numbers of variants. In order to effectively maintain and develop these product variants, Product-Line Engineering methods are often considered, while Model-based Systems Engineering practices are commonly utilized to tackle product complexity. In this paper, we report on an industrial case study concerning the ongoing adoption of Product Line Engineering in the Model-based Systems Engineering environment at Volvo Construction Equipment (Volvo CE) in Sweden. In the study, we identify and define a Product Line Engineering process that is aligned with Model-based Systems Engineering activities at the engines control department of Volvo CE. Furthermore, we discuss the implications of the migration from the current development process to a Model-based Product Line Engineering-oriented process. This process, and its implications, are derived by conducting and analyzing interviews with Volvo CE employees, inspecting artifacts and documents, and by means of participant observation. Based on the results of a first system model iteration, we were able to document how Model-based Systems Engineering and variability modeling will affect development activities, work products and stakeholders of the work products.
|
|
| 30. |
- Causevic, Adnan, et al.
(author)
-
A Survey on Industrial Software Engineering
- 2009
-
In: AGILE PROCESSES IN SOFTWARE ENGINEERING AND EXTREME PROGRAMMING. - Berlin, Heidelberg : Springer. - 9783642018527 ; , s. 240-241
-
Book chapter (peer-reviewed)abstract
- In this paper, we present on-going work on data collected by a questionnaire surveying process practices, preferences, and methods in industrial software engineering.
|
|
| 31. |
- Causevic, Adnan, et al.
(author)
-
An Industrial Survey on Contemporary Aspects of Software Testing
- 2010
-
In: ICST 2010 - 3rd International Conference on Software Testing, Verification and Validation. - Paris, France. - 9780769539904 ; , s. 393-401
-
Conference paper (peer-reviewed)abstract
- Software testing is a major source of expense in software projects and a proper testing process is a critical ingredient in the cost-efficient development of high-quality software. Contemporary aspects, such as the introduction of a more lightweight process, trends towards distributed development, and the rapid increase of software in embedded and safety-critical systems, challenge the testing process in unexpected manners. To our knowledge, there are very few studies focusing on these aspects in relation to testing as perceived by different contributors in the software development process. This paper qualitatively and quantitatively analyses data from an industrial questionnaire survey, with a focus on current practices and preferences on contemporary aspects of software testing. Specifically, the analysis focuses on perceptions of the software testing process in different categories of respondents. Categorization of respondents is based on safety-criticality, agility, distribution of development, and application domain. While confirming some of the commonly acknowledged facts, our findings also reveal notable discrepancies between preferred and actual testing practices. We believe continued research efforts are essential to provide guidelines in the adaptation of the testing process to take care of these discrepancies, thus improving the quality and efficiency of the software development.
|
|
| 32. |
- Causevic, Adnan, et al.
(author)
-
An Industrial Survey on Software Process Practices, Preferences and Methods
- 2009
-
Reports (other academic/artistic)abstract
- This report presents the questionnaire questions and responses of a web-based survey conducted during early spring 2009. It was intended to survey current processes, practices, and methods in the software industry. The report contains no analysis of the data, but is intended to be a reference for other publications, both by ourselves, but by making the data publicly available, also by others.
|
|
| 33. |
- Causevic, Adnan, et al.
(author)
-
Effects of Negative Testing on TDD : An Industrial Experiment
- 2013
-
In: Agile Processes in Software Engineering and Extreme Programming. - Berlin, Heidelberg : Springer. - 9783642383137 - 9783642383144 ; , s. 91-105
-
Book chapter (peer-reviewed)abstract
- In our recent academic experiments, an existence of positive test bias, that is lack of negative test cases, was identified when a test driven development approach was used. At the same time, when defect detecting ability of individual test cases was calculated, it was noted that the probability of a negative test case to detect a defect was substantially higher than that of a positive test case. The goal of this study is to investigate the existence of positive test bias in test driven development within an industrial context, and measure defect detecting ability of both positive and negative test cases. An industrial experiment was conducted at Infosys Ltd. India, whose employees voluntarily signed up to participate in the study and were randomly assigned to groups utilizing test driven development, test driven development with negative testing, and test last development. Source code and test cases created by each participant during the study were collected and analysed. The collected data indicate a statistically significant difference between the number of positive and negative test cases created by industrial participants, confirming the existence of positive test bias. The difference in defect detecting ability of positive and negative test cases is also statistically significant. As a result, similarly to our previous academic study, 29% of all test cases were negative, contributing by revealing as much as 71% of all the defects found by all test cases. With this industrial experiment, we confirmed the existence of a positive test bias in an industrial context, as well as significantly higher defect detecting ability of negative test cases.
|
|
| 34. |
|
|
| 35. |
- Causevic, Adnan, et al.
(author)
-
Factors Limiting Industrial Adoption of Test Driven Development: A Systematic Review
- 2011
-
In: 4th IEEE International Conference on Software Testing, Verification, and Validation, ICST 2011; Berlin; 21 March 2011 through 25 March 2011. - 9780769543420 ; , s. 337-346
-
Conference paper (peer-reviewed)abstract
- Test driven development (TDD) is one of the basic practices of agile software development and both academia and practitioners claim that TDD, to a certain extent, improves the quality of the code produced by developers. However, recent results suggest that this practice is not followed to the extent preferred by industry. In order to pinpoint specific obstacles limiting its industrial adoption we have conducted a systematic literature review on empirical studies explicitly focusing on TDD as well as indirectly addressing TDD. Our review has identified seven limiting factors viz., increased development time, insufficient TDD experience/knowledge, lack of upfront design, domain and tool specific issues, lack of developer skill in writing test cases, insufficient adherence to TDD protocol, and legacy code. The results of this study is of special importance to the testing community, since it outlines the direction for further detailed scientific investigations as well as highlights the requirement of guidelines to overcome these limiting factors for successful industrial adoption of TDD.
|
|
| 36. |
- Causevic, Adnan, et al.
(author)
-
Impact of Test Design Technique Knowledge on Test Driven Development: A Controlled Experiment
- 2012
-
In: Lecture Notes in Business Information Processing, vol. 111 LNBIP. - Berlin, Heidelberg : Springer. - 9783642303494 ; , s. 138-152
-
Book chapter (peer-reviewed)abstract
- Agile development approaches are increasingly being followed and favored by the industry. Test Driven Development (TDD) is a key agile practice and recent research results suggest that the successful adoption of TDD depends on different limiting factors, one of them being insufficient developer testing skills. The goal of this paper is to investigate if developers who are educated on general testing knowledge will be able to utilize TDD more efectively. We conducted a controlled experiment with master students during the course on Software Verification & Validation (V&V) where source code and test cases created by each participant during the labs as well as their answers on a survey questionnaire were collected and analyzed. Descriptive statistics indicate improvements in statement coverage. However, no statistically significant differences could be established between the pre- and post-course groups of students. By qualitative analysis of students’ tests, we noticed a lack of test cases for non-stated requirements ("negative" tests) resulting in a non-detection of bugs. Students did show preference towards TDD in surveys. Although further research is required to fully establish this, we believe that identifying specific testing knowledge which is complementary to the testing skills of a new TDD developer would enable developers to perform their tasks in a more eficient manner.
|
|
| 37. |
|
|
| 38. |
- Causevic, Adnan, et al.
(author)
-
Quality of Testing in Test Driven Development
- 2012
-
In: 2012 8th International Conference on the Quality of Information and Communications Technology, QUATIC 2012. - : IEEE. - 9780769547770 ; , s. 266-271
-
Conference paper (peer-reviewed)abstract
- Test-driven development is an essential part of eXtreme Programming approach with the preference of being followed in other Agile methods as well. For several years, researchers are performing empirical investigations to evaluate quality improvements in the resulting code when test-driven development is being used. However, very little had been reported into investigating the quality of the testing performed in conjunction with test-driven development. In this paper we present results from an experiment specifically designed to evaluate the quality of test cases created by developers who used the test-first and the traditional test-last approaches. On an average, the quality of testing in test-driven development was almost the same as the quality of testing using test-last approach. However, detailed analysis of test cases, created by test-driven development group, revealed that 29% of test cases were "negative" test cases (based on non-specified requirements) but contributing as much as 65% to the overall tests quality score of test-first developers. We are currently investigating the possibility of extending test-driven development to facilitate non-specified requirements to a higher extent and thus minimise the impact of a potentially inherent effect of positive test bias.
|
|
| 39. |
|
|
| 40. |
- Causevic, Adnan, et al.
(author)
-
TDDHQ : Achieving Higher Quality Testing in Test Driven Development
- 2013
-
In: Euromicro Conference on Software Engineering and Advanced Applications. ; , s. 33-36
-
Conference paper (peer-reviewed)abstract
- Test driven development (TDD) appears not to be immune to positive test bias effects, as we observed in several empirical studies. In these studies, developers created a significantly larger set of positive tests, but at the same time the number of defects detected with negative tests is significantly higher than those detected by positive ones. In this paper we propose the concept of TDDHQ which is aimed at achieving higher quality of testing in TDD by augmenting the standard TDD with suitable test design techniques. To exemplify this concept, we present combining equivalence partitioning test design technique together with the TDD, for the purpose of improving design of test cases. Initial evaluation of this approach showed a noticeable improvement in the quality of test cases created by developers utilising TDDHQ approach.
|
|
| 41. |
|
|
| 42. |
- Causevic, Adnan, et al.
(author)
-
Test Case Quality in Test Driven Development: A Study Design and a Pilot Experiment
- 2012
-
In: EASE 2012, Proceedings. - : IET. - 9781849195416 ; , s. 223-227
-
Conference paper (peer-reviewed)abstract
- Background: Test driven development, as a side-effect of developing software, will produce a set of accompanied test cases which can protect implemented features during code refactoring. However, recent research results point out that successful adoption of test driven development might be limited by the testing skills of developers using it. Aim: Main goal of this paper is to investigate if there is a difference between the quality of test cases created while using test-first and test-last approaches. Additional goal of this paper is to measure the code quality produced using test-first and test-last approaches. Method: A pilot study was conducted during the master level course on Software Verification & Validation at Mälardalen University. Students were working individually on the problem implementation by being randomly assigned to a test-first or a test-last (control) group. Source code and test cases created by each participant during the study, as well as their answers on a survey questionnaire after the study, were collected and analysed. The quality of the test cases is analysed from three perspectives: (i) code coverage, (ii) mutation score and (iii) the total number of failing assertions. Results: The total number of test cases with failing assertions (test cases revealing an error in the code) was nearly the same for both test-first and test-last groups. This can be interpreted as "test cases created by test-first developers were as good as (or as bad as) test cases created by test-last developers". On the contrary, solutions created by test-first developers had, on average, 27% less failing assertions when compared to solutions created by the test-last group. Conclusions: Though the study provided some interesting observations, it needs to be conducted as a fully controlled experiment with a higher number of participants in order to validate statistical significance of the presented results.
|
|
| 43. |
|
|
| 44. |
- Derehag, Jesper, et al.
(author)
-
Transitioning Fault Prediction Models to a New Environment
- 2016
-
In: Proceedings - 2016 12th European Dependable Computing Conference, EDCC 2016. - 9781509015825 ; , s. 241-248
-
Conference paper (peer-reviewed)abstract
- We describe the application and evaluation of fault prediction algorithms to a project developed by a Swedish company that transitioned from waterfall to agile development methods. The project used two different version control systems and a separate bug tracking system during its lifetime. The algorithms were originally designed for use on systems implemented with a traditional waterfall process at American companies that maintained their project records in an integrated database system that combined bug recording and version control. We compare the performance of the original prediction model on the American systems to the results obtained in the Swedish environment in both its pre-agile and agile stages. We also consider the impact of additional variables in the model.
|
|
| 45. |
- Ekman, M., et al.
(author)
-
Tool qualification for safety related systems
- 2014
-
In: Ada User Journal. - 1381-6551. ; 35, s. 47-54
-
Journal article (peer-reviewed)abstract
- Tools used in the development of safety related software applications need to be qualified as safe. That is, the tools cannot be allowed to introduce hazardous faults into the application, e.g., a compiler shall not generate dangerous code due to failure of the compiler. In many cases laws and regulations require the product development of safety related applications to comply with industry sector specific safety standards. Examples of such standards include EN50129/50128 for railway applications, ISO/EN13849 for machines with moving parts, DO-178B/C for avionics, or IS026262 for cars. These standards require the use of a rigorous development and maintenance process. The standards are also mainly intended to be used when developing systems from scratch. However, most development and test tools are not developed from scratch according to the rigorous processes of these standards. In order to address this issue, some of the standards provide means for qualifying existing tools as a more lightweight and pragmatic alternative to a regular certification process. In this paper we analyze the concept of these qualification approaches. The result of the analysis in our contribution includes a set of approaches that can be applied individually or as a combination in order to reduce the effort needed for qualifying tools. As a running example we use one of the most flexible but at the same time dangerous, even prohibited, maintenance techniques available: dynamic instrumentation of executing code. With this example, we describe how exceptions in these standards can be utilized in order to qualify a dynamic instrumentation tool with a minimal effort, without following the process of tool certification as defined by the standards.
|
|
| 46. |
- Eldh, Sigrid, et al.
(author)
-
Framework for Comparing Efficiency, Effectiveness and Applicability of Software Testing Techniques
- 2006
-
In: Proceedings - Testing: Academic and Industrial Conference - Practice and Research Techniques, TAIC PART 2006. ; , s. 159-170
-
Conference paper (peer-reviewed)abstract
- Software testing is expensive for the industry, and always constrained by time and effort. Although there is a multitude of test techniques, there are currently no scientifically based guidelines for the selection of appropriate techniques of different domains and contexts. For large complex systems, some techniques are more efficient in finding failures than others and some are easier to apply than others are. From an industrial perspective, it is important to find the most effective and efficient test design technique that is possible to automate and apply. In this paper, we propose an experimental framework for comparison of test techniques with respect to efficiency, effectiveness and applicability. We also plan to evaluate ease of automation, which has not been addressed by previous studies. We highlight some of the problems of evaluating or comparingtest techniques in an objective manner. We describe our planned process for this multi-phase experimental study. This includes presentation of some of the important measurements to be collected with the dual goals of analyzing the properties of the test technique, as well as validating our experimental framework.
|
|
| 47. |
- Eldh, Sigrid, 1961-, et al.
(author)
-
Robustness Testing of Mobile Telecommunication Systems : A Case Study on Industrial Practice and Challenges
- 2012
-
In: Proceedings - IEEE 5th International Conference on Software Testing, Verification and Validation, ICST 2012. - : IEEE Press. - 9780769546704 ; , s. 895-900
-
Conference paper (peer-reviewed)abstract
- Robustness relates to the capability of a system to handle internal and external negative situations and disturbances. Robustness testing is the act of subjecting the system under test to such disturbances in a controlled manner. The objective of this study is to understand how robustness is considered in the development and testing of large-scale telecom systems, to identify the main challenges related to robustness and robustness testing, and to identify potential improvements to the current situation. We performed an exploratory case study of a telecom industry. Data was collected through interviews, study of documentation, and participant observation. Our result is a number of challenges related to robustness testing. The key challenge identified relates to understanding how robustness test can be broken down from a system-level to a low-level perspective. Our conclusion is that the area of robustness test is challenging for large complex systems, where the understanding of how to provoke complex failures and derive a root cause, as well as defining the correct level of robust design in software is not sufficiently explored.
|
|
| 48. |
|
|
| 49. |
- Enoiu, Eduard, 1986-
(author)
-
Automatic test generation for industrial control software
- 2016
-
Doctoral thesis (other academic/artistic)abstract
- Since the early days of software testing, automatic test generation has been suggested as a way of allowing tests to be created at a lower cost. However, industrially useful and applicable tools for automatic test generation are still scarce. As a consequence, the evidence regarding the applicability or feasibility of automatic test generation in industrial practice is limited. This is especially problematic if we consider the use of automatic test generation for industrial safety-critical control systems, such as are found in power plants, airplanes, or trains.In this thesis, we improve the current state of automatic test generation by developing a technique based on model-checking that works with IEC 61131-3 industrial control software. We show how automatic test generation for IEC 61131-3 programs, containing both functional and timing information, can be solved as a model checking problem for both code and mutation coverage criteria. The developed technique has been implemented in the CompleteTest tool. To evaluate the potential application of our technique, we present several studies where the tool is applied to industrial control software. Results show that CompleteTest is viable for use in industrial practice; it is efficient in terms of the time required to generate tests that satisfy both code and mutation coverage and scales well for most of the industrial programs considered.However, our results also show that there are still challenges associated with the use of automatic test generation. In particular, we found that while automatically generated tests, based on code coverage, can exercise the logic of the software as well as tests written manually, and can do so in a fraction of the time, they do not show better fault detection compared to manually created tests. Specifically, it seems that manually created tests are able to detect more faults of certain types (i.e, logical replacement, negation insertion and timer replacement) than automatically generated tests. To tackle this issue, we propose an approach for improving fault detection by using mutation coverage as a test criterion. We implemented this approach in the CompleteTest tool and used it to evaluate automatic test generation based on mutation testing. While the resulting tests were more effective than automatic tests generated based on code coverage, in terms of fault detection, they still were not better than manually created tests.In summary, our results highlight the need for improving the goals used by automatic test generation tools. Specifically, fault detection scores could be increased by considering some new mutation operators as well as higher-order mutations. Our thesis suggests that automatically generated test suites are significantly less costly in terms of testing time than manually created test suites. One conclusion, strongly supported by the results of this thesis, is that automatic test generation is efficient but currently not quite as effective as manual testing. This is a significant progress that needs to be further studied; we need to consider the implications and the extent to which automatic test generation can be used in the development of reliable safety-critical systems.
|
|
| 50. |
- Enoiu, Eduard Paul, et al.
(author)
-
A Comparative Study of Manual and Automated Testing for Industrial Control Software
- 2017
-
In: Proceedings - 10th IEEE International Conference on Software Testing, Verification and Validation, ICST 2017. - 9781509060313 ; , s. 412-417
-
Conference paper (peer-reviewed)abstract
- Automated test generation has been suggested as a way of creating tests at a lower cost. Nonetheless, it is not very well studied how such tests compare to manually written ones in terms of cost and effectiveness. This is particularly true for industrial control software, where strict requirements on both specification-based testing and code coverage typically are met with rigorous manual testing. To address this issue, we conducted a case study in which we compared manually and automatically created tests. We used recently developed real-world industrial programs written in the IEC 61131-3, a popular programming language for developing industrial control systems using programmable logic controllers. The results show that automatically generated tests achieve similar code coverage as manually created tests, but in a fraction of the time (an average improvement of roughly 90%). We also found that the use of an automated test generation tool does not result in better fault detection in terms of mutation score compared to manual testing. Specifically, manual tests more effectively detect logical, timer and negation type of faults, compared to automatically generated tests. The results underscore the need to further study how manual testing is performed in industrial practice and the extent to which automated test generation can be used in the development of reliable systems.
|
|
