| 1. |
- Flemström, Daniel, et al.
(författare)
-
A Research Roadmap for Test Design in Automated Integration Testing of Vehicular Systems
- 2016. - 9
-
Ingår i: FASSI 2016. - : International Academy, Research and Industry Association (IARIA). - 9781612084978 ; , s. 18-23
-
Konferensbidrag (refereegranskat)abstract
- An increasing share of the innovations emerging in the vehicular industry are implemented in software. Consequently, vehicular electrical systems are becoming more and more complex with an increasing number of functions, computational nodes and complex sensors, e.g., cameras and radars. The introduction of autonomous functional components, such as advanced driver assistance systems, highlight the foreseeable complexity of different parts of the system interacting with each other and with the human driver. It is of utmost importance that the testing effort can scale with this increasing complexity. In this paper, we review the challenges that we are facing in integration testing of complex embedded vehicular systems. Further, based on these challenges we outline a set of research directions for semi-automated or automated test design and execution in integration testing of vehicular systems. While the discussion is exemplified with our hands-on experience of the automotive industry, much of the concepts can be generalised to a broader setting of complex embedded systems.
|
|
| 2. |
- Flemström, Daniel, et al.
(författare)
-
Exploring Test Overlap in System Integration : An Industrial Case Study
- 2016
-
Ingår i: 42nd Euromicro Conference series on Software Engineering and Advanced Applications SEAA 2016. - 9781509028191 ; , s. 303-312
-
Konferensbidrag (refereegranskat)abstract
- Tougher safety regulations, global competition and ever increasing complexity of embedded software puts extensive pressure on the effectiveness of the software testing procedures. Previous studies have found that there exist overlaps (i.e., multiple instances of highly similar test cases) and even redundancies in the software testing process. Such overlap has been found between versions, variants and integration levels, but primarily at unit test level. Given large embedded systems involving many subsystems, does overlap exist within the system integration testing as well? In this paper, we present an industrial case study, aiming to a) evaluate if there exist test overlaps within the given context, b) if so, investigate how these overlaps are distributed, and c) find ways of reducing test effort by investigating how the knowledge of overlaps and their distribution may be used for finding candidate test cases for automation, maintenance or even removal. We have studied manual test cases, written in natural language, at a large vehicular manufacturer in Sweden. In particular, we have collected and analyzed test cases from the system integration testing levels of four different projects of a vehicle control management system. Using a similarity function, we evaluate if any overlaps between test cases exist, and where. We found that overlaps do exist within the system integration level, particularly in the form of partial test step sequences. However, very few test cases overlapped in their entirety. Some candidates for test step automation and update propagation were identified, but none for easy removal.
|
|
| 3. |
- Flemström, Daniel, et al.
(författare)
-
From natural language requirements to passive test cases using guarded assertions
- 2018
-
Ingår i: Proceedings - 2018 IEEE 18th International Conference on Software Quality, Reliability, and Security, QRS 2018. - : Institute of Electrical and Electronics Engineers Inc.. - 9781538677575 ; , s. 470-481
-
Konferensbidrag (refereegranskat)abstract
- In large-scale embedded system development, requirements are often expressed in natural language. Translating these requirements to executable test cases, while keeping the test cases and requirements aligned, is a challenging task. While such a transformation typically requires extensive domain knowledge, we show that a systematic process in combination with passive testing would facilitate the translation as well as linking the requirements to tests. Passive testing approaches observe the behavior of the system and test their correctness without interfering with the normal behavior. We use a specific approach to passive testing: guarded assertions (G/A). This paper presents a method for transforming system requirements expressed in natural language into G/As. We further present a proof of concept evaluation, performed at Bombardier Transportation Sweden AB, in which we show how the process would be used, together with practical advice of the reasoning behind the translation steps.
|
|
| 4. |
- Flemström, Daniel, 1971-
(författare)
-
Industrial System Level Test Automation
- 2021
-
Doktorsavhandling (övrigt vetenskapligt/konstnärligt)abstract
- Vehicular software systems control and monitor many safety-critical functions, such as automated emergency brakes and anti-spin. These functions are integrated and tested at system level to ensure that the entire system works as intended.Traditionally, these functions or selected combinations of functions are tested in isolation. Although desired, rigorous testing of combinations of functions prior to deployment is seldom possible. One reason is the overwhelming work required to write new test cases for the nearly infinite combinations of functions and driving scenarios. Since software testing already accounts for up to 60% of the software development cost and the execution time in test rigs is expensive, further testing must be achieved by running test cases more in parallel. Moreover, new test cases must be reusable in different driving scenarios to cover more combinations of functions. One solution is to express the test logic in a new way so that it can be executed in parallel, independently of other tests and independent of the input stimuli. This would allow reusing the test logic for different sequences or drive scenarios. Passive testing is one such approach that has not yet been used much for vehicular software due to perceived difficulties, although it is well-established in other domains. One particular problem is how to specify and execute passive test cases for vehicular systems in an, for the test engineer, intuitive and straightforward manner. Further, there is a lack of tool support and knowledge on efficiently applying passive testing in an industrial context. Thus, the overall research goal of this thesis is to propose and evaluate industrially applicable methods and tools for passive testing at the system level of vehicular software systems. The research is based on a series of papers and case studies within the vehicular industry. In contrast to existing specification languages based on formal mathematical expressions, considerable effort was spent creating an intuitive language and an interactive tool, simple but powerful enough, to encourage the industrial application of passive testing. The main contributions include an easy-to-write and easy-to-read description language for passive test cases, an interactive development environment, and knowledge on how to succeed in passive testing in an industrial software development process. The findings of this thesis contribute to making passive testing a viable method for system-level testing and a way of reusing test logic. In the case of a studied train control management system, using passive testing may reuse up to 50% of the test logic for the safety-related requirements and 10% of the non-safety-related requirements.
|
|
| 5. |
- Flemström, Daniel, et al.
(författare)
-
Similarity-based prioritization of test case automation
- 2018
-
Ingår i: Software quality journal. - : Springer Science and Business Media LLC. - 0963-9314 .- 1573-1367. ; 26:4, s. 1421-1449
-
Tidskriftsartikel (refereegranskat)abstract
- The importance of efficient software testing procedures is driven by an ever increasing system complexity as well as global competition. In the particular case of manual test cases at the system integration level, where thousands of test cases may be executed before release, time must be well spent in order to test the system as completely and as efficiently as possible. Automating a subset of the manual test cases, i.e, translating the manual instructions to automatically executable code, is one way of decreasing the test effort. It is further common that test cases exhibit similarities, which can be exploited through reuse when automating a test suite. In this paper, we investigate the potential for reducing test effort by ordering the test cases before such automation, given that we can reuse already automated parts of test cases. In our analysis, we investigate several approaches for prioritization in a case study at a large Swedish vehicular manufacturer. The study analyzes the effects with respect to test effort, on four projects with a total of 3919 integration test cases constituting 35,180 test steps, written in natural language. The results show that for the four projects considered, the difference in expected manual effort between the best and the worst order found is on average 12 percentage points. The results also show that our proposed prioritization method is nearly as good as more resource demanding meta-heuristic approaches at a fraction of the computational time. Based on our results, we conclude that the order of automation is important when the set of test cases contain similar steps (instructions) that cannot be removed, but are possible to reuse. More precisely, the order is important with respect to how quickly the manual test execution effort decreases for a set of test cases that are being automated.
|
|
| 6. |
- Flemström, Daniel, 1971-
(författare)
-
Similarity-Based Test Effort Reduction
- 2017
-
Licentiatavhandling (övrigt vetenskapligt/konstnärligt)abstract
- Embedded computer systems are all around us. We find them in everything, from dishwashers to cars and airplanes. They must always work correctly and moreover, often within certain time constraints. The software of such a system can be very large and complex, e.g. in the case of a car or a train. Hence, we develop the software for embedded systems in smaller, manageable, parts. These parts can be successively integrated, until they form the complete software for the embedded system, possibly at different levels. This phase of the development process is called the system integration phase and is one of the most critical phases in the development of embedded systems. In this phase, substantial effort is spent on testing activities.Studies have found that a considerable amount of test effort is wasteful due to people, unknowingly or by necessity, performing similar (or even overlapping) test activities. Consequently, test cases may end up to be similar, partially or wholly. We identified such test similarities in a case study of 2500 test cases, written in natural language, from four different projects in the embedded vehicular domain. Such information can be used for reducing effort when maintaining or automating similar test cases.In another case study in the same domain, we investigated several approaches for prioritizing test cases to automate with the objective to reduce manual test effort as quick as possible given that similar automated tests could be reused (similarity-based reuse). We analyzed how the automation order affects the test effort for four projects with a total of 3919 integration test cases, written in natural language. The results showed that similarity-based reuse of automated test case script code, and the best-performing automation order can reduce the expected manual test effort with 20 percentage points.Another way of reducing test effort is to reuse test artifacts from one level of integration to another, instead of duplicating them. We studied such reuse methods, that we denote vertical reuse, in a systematic mapping study. While the results from of our systematic mapping study showed the viability of vertical test reuse methods, our industrial case studies showed that keeping track of similarities and test overlaps is both possible and feasible for test effort reduction. We further conclude that the test case automation order affects the manual test execution effort when there exist similar steps that cannot be removed, but are possible to reuse with respect to test script code.
|
|
| 7. |
- Flemström, Daniel, et al.
(författare)
-
Vertical Test Reuse for Embedded Systems : A Systematic Mapping Study
- 2015. - 11
-
Ingår i: 2015 41st Euromicro Conference on Software Engineering and Advanced Applications. - : Conference Publishing Services. - 9781467375856 ; , s. 317-324
-
Konferensbidrag (refereegranskat)abstract
- Vertical test reuse refers to the the reuse of test cases or other test artifacts over different integration levels in the software or system engineering process. Vertical test reuse has previously been proposed for reducing test effort and improving test effectiveness, particularly for embedded system development. The goal of this study is to provide an overview of the state of the art in the field of vertical test reuse for embedded system development. For this purpose, a systematic mapping study has been performed, identifying 11 papers on vertical test reuse for embedded systems. The primary result from the mapping is a classification of published work on vertical test reuse in the embedded system domain, covering motivations for reuse, reuse techniques, test levels and reusable test artifacts considered, and to what extent the effects of reuse have been evaluated.
|
|
| 8. |
- Flemström, Daniel, et al.
(författare)
-
A Case Study of Interactive Development of Passive Tests
- 2018
-
Ingår i: 5th International Workshop on Requirements Engineering and Testing RET'18. - New York, NY, USA : ACM. ; , s. 13-20
-
Konferensbidrag (övrigt vetenskapligt/konstnärligt)abstract
- Testing in the active sense is the most common way to performverification and validation of systems, but testing in the passivesense has one compelling property: independence. Independencefrom test stimuli and other passive tests opens up for parallel testingand off-line analysis. However, the tests can be difficult to developsince the complete testable state must be expressed using someformalism. We argue that a carefully chosen language togetherwith an interactive work flow, providing immediate feedback, canenable testers to approach passive testing. We have conducted a casestudy in the automotive domain, interviewing experienced testers.The testers have been introduced to, and had hands-on practicewith a tool. The tool is based on Easy Approach to RequirementsSyntax (EARS) and provides an interactive work flow for developingand evaluating test results. The case study shows that i) the testersbelieve passive testing is useful for many of their tests, ii) they seebenefits in parallelism and off-line analysis, iii) the interactive workflow is necessary for writing the testable state expression, but iv)when the testable state becomes too complex, then the proposedlanguage is a limitation. However, the language contributes toconcise tests, resembling executable requirements.
|
|
| 9. |
- Flemström, Daniel, et al.
(författare)
-
Industrial Scale Passive Testing with T-EARS
- 2021
-
Ingår i: Proceedings. ; , s. 351-361
-
Konferensbidrag (refereegranskat)abstract
- Passive testing continuously observes the system or system execution logs without any interference or instrumentation to test diverse combinations of functions, resulting in a more thorough evaluation over time. However, reaching a working solution to passive testing is not without challenges. While there have been some efforts to extract information from system requirements to create passive test cases, to our knowledge, no such efforts are mature enough to be applied in a real, industrial safety-critical context. Our passive testing approach uses the Timed - Easy Approach to Requirements Syntax (T-EARS) specification language and its accompanying tool-chain. This study reports challenges and solutions to introducing system-level passive testing for a vehicular safety-critical system through industrial data analysis, including 116 safety-related requirements. Our results show that passive testing using the T-EARS language and its tool-chain can be used for system-level testing in an industrial setting for 64% of the studied requirements. We identified several sources of false positive results and show how to tune test cases to reduce such false positives systematically. Finally, we show the requirement coverage achieved by a manual test session and that passive testing using T-EARS can find a set of injected faults that are considered hard to find with other test techniques.
|
|
| 10. |
- Flemström, Daniel, et al.
(författare)
-
SAGA Toolbox : Interactive Testing of Guarded Assertions
- 2017
-
Ingår i: Proceedings - 10th IEEE International Conference on Software Testing, Verification and Validation, ICST 2017. - 9781509060313 ; , s. 516-523
-
Konferensbidrag (refereegranskat)abstract
- This paper presents the SAGA toolbox. It centers around development of tests, and analysis of test results, on Guarded Assertions (GA) format. Such a test defines when to test, and what to expect in such a state. The SAGA toolbox lets the user describe the test, and at the same time get immediate feedback on the test result based on a trace from the System Under Test (SUT). The feedback is visual using plots of the trace. This enables the test engineer to play around with the data and use an agile development method, since the data is already there. Moreover, the SAGA toolbox also enables the test engineer to change test stimuli plots to study the effect they have on a test. It can later generate computer programs that can feed these test stimuli to the SUT. This enables an interactive feedback loop, where immediate feedback on changes to the test, or to the test stimuli, indicate whether the test is correct and it passed or failed.
|
|
| 11. |
- Flemström, Daniel, 1971-, et al.
(författare)
-
Specification of Passive Test Cases using an Improved T-EARS Language
-
Annan publikation (övrigt vetenskapligt/konstnärligt)abstract
- Test cases that only observe the system under test can improve parallelism and detection of faults occurring due to unanticipated feature interactions. Traditionally, such \textit{passive} test cases have been challenging to express, partly due to the use of complex mathematical notations. The T-EARS (Timed Easy Approach to Requirements Syntax) language prototype was introduced to respond to this and has received positive feedback from practitioners. However, the prototype suffered from few deficiencies, such as allowing non-intuitive combinations of expressions and usage of temporal specifiers that quickly got difficult to understand. This paper builds on the T-EARS prototype and input from experienced testers on a previous iteration of the language. The collected experience was applied to a new prototype using a structured update process, including a set of system-level requirements from a vehicular software system. The results include a new, improved grammar for the T-EARS language and a description of the evaluation semantics.
|
|
| 12. |
- Flemström, Daniel, et al.
(författare)
-
Specification of Passive Test Cases Using an Improved T-EARS Language
- 2022
-
Ingår i: SOFTWARE QUALITY. - Cham : Springer Science and Business Media Deutschland GmbH. - 9783031041143 ; , s. 63-83
-
Konferensbidrag (refereegranskat)abstract
- Test cases that only observe the system under test can improve parallelism and detection of faults occurring due to unanticipated feature interactions. Traditionally, such passive test cases have been challenging to express, partly due to the use of complex mathematical notations. The T-EARS (Timed Easy Approach to Requirements Syntax) language prototype was introduced to respond to this and has received positive feedback from practitioners. However, the prototype suffered from few deficiencies, such as allowing non-intuitive combinations of expressions and usage of temporal specifiers that quickly got difficult to understand. This paper builds on the T-EARS prototype and input from experienced testers on a previous iteration of the language. The collected experience was applied to a new prototype using a structured update process, including a set of system-level requirements from a vehicular software system. The results include a new, improved grammar for the T-EARS language and a description of the evaluation semantics.
|
|
| 13. |
- Flemström, Daniel
(författare)
-
Tech. Report: Similarity Function Evaluation
- 2017
-
Rapport (övrigt vetenskapligt/konstnärligt)abstract
- This report presents details on and an in-depth evaluation of a similarity function used for detecting similar test steps in manual test cases, written in natural language. Using an industrial data set of 65 000 test steps, we show that even though the similarity function builds on standard functions from the open source data base Postgres, it is capable of finding similarities in parity of what the state of the art suggests. Rather few miss classifications were found. We also show that by fine tuning the function, the number of clusters of similar can be reduced by 13%. Manual inspection further shows that there is potential to reduce the set of clusters even more.
|
|
| 14. |
- Larsson, Thomas, et al.
(författare)
-
Improving Introductory Programming Courses by Using a Simple Accelerated Graphics Library
- 2007
-
Ingår i: SIGRAD 2007. - 9789173939904 ; , s. 24-30
-
Konferensbidrag (refereegranskat)abstract
- We present a platform independent and hardware accelerated graphics library, which has been found to be a suitable educational tool for novice programmers. The purpose of the library is to change the nature of the assignments and projects used in introductory programming courses to something that fascinate and stimulate the students, e.g., game creation. We describe our experiences from using the presented graphics library in four different course instances of our introductory C++ course. The course surveys show that most students found the approach interesting and fun. As faculty, we could clearly see how many students became highly engaged in their projects and some of them accomplished solutions way beyond our expectations. In particular, compared to the programming courses we have given in the past, in which a standard framework for creating window applications was used, we have noticed a significant improvement in terms of the quality of the students' project solutions.
|
|
| 15. |
- Lüders, Frank, et al.
(författare)
-
A Prototype Tool for Software Component Services in Embedded Real-Time Systems
- 2006
-
Ingår i: Component-Based Software Engineering. - Berlin, Heidelberg : Springer. - 3540356282 ; , s. 222-237
-
Bokkapitel (refereegranskat)abstract
- The use of software component models has become popular during the last decade, in particular in the development of software for desktop applications and distributed information systems. However, such models have not been widely used in the domain of embedded real-time systems. There is a considerable amount of research on component models for embedded real-time systems, or even narrower application domains, which focuses on source code components and statically configured systems. This paper explores an alternative approach by laying the groundwork for a component model based on binary components and targeting the broader domain of embedded real-time systems. The work is inspired by component models for the desktop and information systems domains in the sense that a basic component model is extended with a set of services for the targeted application domain. A prototype tool for supporting these services is presented and its use illustrated by a control application.
|
|
| 16. |
|
|
