| 1. |
- Flemström, Daniel, 1971-
(författare)
-
Industrial System Level Test Automation
- 2021
-
Doktorsavhandling (övrigt vetenskapligt/konstnärligt)abstract
- Vehicular software systems control and monitor many safety-critical functions, such as automated emergency brakes and anti-spin. These functions are integrated and tested at system level to ensure that the entire system works as intended.Traditionally, these functions or selected combinations of functions are tested in isolation. Although desired, rigorous testing of combinations of functions prior to deployment is seldom possible. One reason is the overwhelming work required to write new test cases for the nearly infinite combinations of functions and driving scenarios. Since software testing already accounts for up to 60% of the software development cost and the execution time in test rigs is expensive, further testing must be achieved by running test cases more in parallel. Moreover, new test cases must be reusable in different driving scenarios to cover more combinations of functions. One solution is to express the test logic in a new way so that it can be executed in parallel, independently of other tests and independent of the input stimuli. This would allow reusing the test logic for different sequences or drive scenarios. Passive testing is one such approach that has not yet been used much for vehicular software due to perceived difficulties, although it is well-established in other domains. One particular problem is how to specify and execute passive test cases for vehicular systems in an, for the test engineer, intuitive and straightforward manner. Further, there is a lack of tool support and knowledge on efficiently applying passive testing in an industrial context. Thus, the overall research goal of this thesis is to propose and evaluate industrially applicable methods and tools for passive testing at the system level of vehicular software systems. The research is based on a series of papers and case studies within the vehicular industry. In contrast to existing specification languages based on formal mathematical expressions, considerable effort was spent creating an intuitive language and an interactive tool, simple but powerful enough, to encourage the industrial application of passive testing. The main contributions include an easy-to-write and easy-to-read description language for passive test cases, an interactive development environment, and knowledge on how to succeed in passive testing in an industrial software development process. The findings of this thesis contribute to making passive testing a viable method for system-level testing and a way of reusing test logic. In the case of a studied train control management system, using passive testing may reuse up to 50% of the test logic for the safety-related requirements and 10% of the non-safety-related requirements.
|
|
| 2. |
- Flemström, Daniel, 1971-
(författare)
-
Similarity-Based Test Effort Reduction
- 2017
-
Licentiatavhandling (övrigt vetenskapligt/konstnärligt)abstract
- Embedded computer systems are all around us. We find them in everything, from dishwashers to cars and airplanes. They must always work correctly and moreover, often within certain time constraints. The software of such a system can be very large and complex, e.g. in the case of a car or a train. Hence, we develop the software for embedded systems in smaller, manageable, parts. These parts can be successively integrated, until they form the complete software for the embedded system, possibly at different levels. This phase of the development process is called the system integration phase and is one of the most critical phases in the development of embedded systems. In this phase, substantial effort is spent on testing activities.Studies have found that a considerable amount of test effort is wasteful due to people, unknowingly or by necessity, performing similar (or even overlapping) test activities. Consequently, test cases may end up to be similar, partially or wholly. We identified such test similarities in a case study of 2500 test cases, written in natural language, from four different projects in the embedded vehicular domain. Such information can be used for reducing effort when maintaining or automating similar test cases.In another case study in the same domain, we investigated several approaches for prioritizing test cases to automate with the objective to reduce manual test effort as quick as possible given that similar automated tests could be reused (similarity-based reuse). We analyzed how the automation order affects the test effort for four projects with a total of 3919 integration test cases, written in natural language. The results showed that similarity-based reuse of automated test case script code, and the best-performing automation order can reduce the expected manual test effort with 20 percentage points.Another way of reducing test effort is to reuse test artifacts from one level of integration to another, instead of duplicating them. We studied such reuse methods, that we denote vertical reuse, in a systematic mapping study. While the results from of our systematic mapping study showed the viability of vertical test reuse methods, our industrial case studies showed that keeping track of similarities and test overlaps is both possible and feasible for test effort reduction. We further conclude that the test case automation order affects the manual test execution effort when there exist similar steps that cannot be removed, but are possible to reuse with respect to test script code.
|
|
| 3. |
- Flemström, Daniel, 1971-, et al.
(författare)
-
Specification of Passive Test Cases using an Improved T-EARS Language
-
Annan publikation (övrigt vetenskapligt/konstnärligt)abstract
- Test cases that only observe the system under test can improve parallelism and detection of faults occurring due to unanticipated feature interactions. Traditionally, such \textit{passive} test cases have been challenging to express, partly due to the use of complex mathematical notations. The T-EARS (Timed Easy Approach to Requirements Syntax) language prototype was introduced to respond to this and has received positive feedback from practitioners. However, the prototype suffered from few deficiencies, such as allowing non-intuitive combinations of expressions and usage of temporal specifiers that quickly got difficult to understand. This paper builds on the T-EARS prototype and input from experienced testers on a previous iteration of the language. The collected experience was applied to a new prototype using a structured update process, including a set of system-level requirements from a vehicular software system. The results include a new, improved grammar for the T-EARS language and a description of the evaluation semantics.
|
|
