| 1. |
- Flemström, Daniel, 1971-
(författare)
-
Industrial System Level Test Automation
- 2021
-
Doktorsavhandling (övrigt vetenskapligt/konstnärligt)abstract
- Vehicular software systems control and monitor many safety-critical functions, such as automated emergency brakes and anti-spin. These functions are integrated and tested at system level to ensure that the entire system works as intended.Traditionally, these functions or selected combinations of functions are tested in isolation. Although desired, rigorous testing of combinations of functions prior to deployment is seldom possible. One reason is the overwhelming work required to write new test cases for the nearly infinite combinations of functions and driving scenarios. Since software testing already accounts for up to 60% of the software development cost and the execution time in test rigs is expensive, further testing must be achieved by running test cases more in parallel. Moreover, new test cases must be reusable in different driving scenarios to cover more combinations of functions. One solution is to express the test logic in a new way so that it can be executed in parallel, independently of other tests and independent of the input stimuli. This would allow reusing the test logic for different sequences or drive scenarios. Passive testing is one such approach that has not yet been used much for vehicular software due to perceived difficulties, although it is well-established in other domains. One particular problem is how to specify and execute passive test cases for vehicular systems in an, for the test engineer, intuitive and straightforward manner. Further, there is a lack of tool support and knowledge on efficiently applying passive testing in an industrial context. Thus, the overall research goal of this thesis is to propose and evaluate industrially applicable methods and tools for passive testing at the system level of vehicular software systems. The research is based on a series of papers and case studies within the vehicular industry. In contrast to existing specification languages based on formal mathematical expressions, considerable effort was spent creating an intuitive language and an interactive tool, simple but powerful enough, to encourage the industrial application of passive testing. The main contributions include an easy-to-write and easy-to-read description language for passive test cases, an interactive development environment, and knowledge on how to succeed in passive testing in an industrial software development process. The findings of this thesis contribute to making passive testing a viable method for system-level testing and a way of reusing test logic. In the case of a studied train control management system, using passive testing may reuse up to 50% of the test logic for the safety-related requirements and 10% of the non-safety-related requirements.
|
|
| 2. |
- Flemström, Daniel, 1971-
(författare)
-
Similarity-Based Test Effort Reduction
- 2017
-
Licentiatavhandling (övrigt vetenskapligt/konstnärligt)abstract
- Embedded computer systems are all around us. We find them in everything, from dishwashers to cars and airplanes. They must always work correctly and moreover, often within certain time constraints. The software of such a system can be very large and complex, e.g. in the case of a car or a train. Hence, we develop the software for embedded systems in smaller, manageable, parts. These parts can be successively integrated, until they form the complete software for the embedded system, possibly at different levels. This phase of the development process is called the system integration phase and is one of the most critical phases in the development of embedded systems. In this phase, substantial effort is spent on testing activities.Studies have found that a considerable amount of test effort is wasteful due to people, unknowingly or by necessity, performing similar (or even overlapping) test activities. Consequently, test cases may end up to be similar, partially or wholly. We identified such test similarities in a case study of 2500 test cases, written in natural language, from four different projects in the embedded vehicular domain. Such information can be used for reducing effort when maintaining or automating similar test cases.In another case study in the same domain, we investigated several approaches for prioritizing test cases to automate with the objective to reduce manual test effort as quick as possible given that similar automated tests could be reused (similarity-based reuse). We analyzed how the automation order affects the test effort for four projects with a total of 3919 integration test cases, written in natural language. The results showed that similarity-based reuse of automated test case script code, and the best-performing automation order can reduce the expected manual test effort with 20 percentage points.Another way of reducing test effort is to reuse test artifacts from one level of integration to another, instead of duplicating them. We studied such reuse methods, that we denote vertical reuse, in a systematic mapping study. While the results from of our systematic mapping study showed the viability of vertical test reuse methods, our industrial case studies showed that keeping track of similarities and test overlaps is both possible and feasible for test effort reduction. We further conclude that the test case automation order affects the manual test execution effort when there exist similar steps that cannot be removed, but are possible to reuse with respect to test script code.
|
|
| 3. |
- Bilic, Damir
(författare)
-
Managing Variability in SysML Models of Automotive Systems
- 2020
-
Licentiatavhandling (övrigt vetenskapligt/konstnärligt)abstract
- Organizations developing software-intensive systems inevitably face increasing complexity of developed products, mainly due to rapid advancements in all domains of technology. Many such organizations are considering model-based systems engineering (MBSE) practices to cope with the increasing complexity. The use of models, as a central role during product design, promises to provide benefits such as enhanced communication among system stakeholders, continuous verification, improved design integrity, traceability between requirements and system artifacts and many more. Additionally, products are often built in many variants. That is especially obvious in the automotive domain, where customers have the ability to configure vehicles with hundreds of configuration options. To deal with the variability, a product line engineering approach is often used. It allows the development of a family of similar software-intensive systems that share a common base while being adapted to individual customer requirements.In this thesis, the overall goal is to evaluate and facilitate the combination of the two mentioned approaches, model-based systems engineering and product line engineering, in an industrial environment. To achieve the main thesis goal, it was divided into three separate research goals.The first goal was to identify challenges when applying an annotation-based approach for variant management in SysML models on a use case provided by Volvo Construction Equipment. The aim was to identify and understand challenges when using existing tool support to manage variants in implementation artifacts of existing products. The second research goal was to identify reuse-related challenges in the ``clone-and-own'' based development process of Volvo CE. Moreover, we assess the effects of model-based product line engineering on the identified challenges. Lastly, the third research goal was to develop an approach for consistency checking between variability- and SysML system models. To achieve that, we develop an integrated tool chain for model-based product line engineering that allows the integration of variable artifacts, which are not documented in system models, into the development process. Secondly, we define and develop an approach for consistency checking between variability models that describe the system in terms of features and implementation models that describe how variability is implemented in the product itself, since such support does not exist in current state of the art tools.In conclusion, based on the results from the results of case studies at Volvo CE, it was shown that model-based product line engineering has the potential to improve communication and highlight implications of variability to stakeholders (e.g. to non-technical staff), improve traceability between variability in requirements and variability in design and implementation, improve consistency through constraints between variants and automate repetitive activities.In other words, it shows potential for improving product quality while reducing the development lead time. However, the evaluation and measurement of improvement will be left for future work as measuring the product quality and lead time requires an organizational roll out of model-based product-line engineering.
|
|
| 4. |
|
|
| 5. |
- Enoiu, Eduard, 1986-
(författare)
-
Automatic test generation for industrial control software
- 2016
-
Doktorsavhandling (övrigt vetenskapligt/konstnärligt)abstract
- Since the early days of software testing, automatic test generation has been suggested as a way of allowing tests to be created at a lower cost. However, industrially useful and applicable tools for automatic test generation are still scarce. As a consequence, the evidence regarding the applicability or feasibility of automatic test generation in industrial practice is limited. This is especially problematic if we consider the use of automatic test generation for industrial safety-critical control systems, such as are found in power plants, airplanes, or trains.In this thesis, we improve the current state of automatic test generation by developing a technique based on model-checking that works with IEC 61131-3 industrial control software. We show how automatic test generation for IEC 61131-3 programs, containing both functional and timing information, can be solved as a model checking problem for both code and mutation coverage criteria. The developed technique has been implemented in the CompleteTest tool. To evaluate the potential application of our technique, we present several studies where the tool is applied to industrial control software. Results show that CompleteTest is viable for use in industrial practice; it is efficient in terms of the time required to generate tests that satisfy both code and mutation coverage and scales well for most of the industrial programs considered.However, our results also show that there are still challenges associated with the use of automatic test generation. In particular, we found that while automatically generated tests, based on code coverage, can exercise the logic of the software as well as tests written manually, and can do so in a fraction of the time, they do not show better fault detection compared to manually created tests. Specifically, it seems that manually created tests are able to detect more faults of certain types (i.e, logical replacement, negation insertion and timer replacement) than automatically generated tests. To tackle this issue, we propose an approach for improving fault detection by using mutation coverage as a test criterion. We implemented this approach in the CompleteTest tool and used it to evaluate automatic test generation based on mutation testing. While the resulting tests were more effective than automatic tests generated based on code coverage, in terms of fault detection, they still were not better than manually created tests.In summary, our results highlight the need for improving the goals used by automatic test generation tools. Specifically, fault detection scores could be increased by considering some new mutation operators as well as higher-order mutations. Our thesis suggests that automatically generated test suites are significantly less costly in terms of testing time than manually created test suites. One conclusion, strongly supported by the results of this thesis, is that automatic test generation is efficient but currently not quite as effective as manual testing. This is a significant progress that needs to be further studied; we need to consider the implications and the extent to which automatic test generation can be used in the development of reliable safety-critical systems.
|
|
| 6. |
- Sundmark, Daniel, 1975-
(författare)
-
Structural System-Level Testing of Embedded Real-Time Systems
- 2008
-
Doktorsavhandling (övrigt vetenskapligt/konstnärligt)abstract
- People make mistakes. Software engineers are no exception to this fact. When software engineers make mistakes, these manifest in the form of buggy software - a major problem in today's industry. The existence of bugs is commonly detected using testing, the process of executing the software and checking if its behaviour complies with the specification. As limitations in time make testing of the entire software behaviour impracticable, testers need to make informed decisions on how to test the software to detect as many bugs as possible.In the realm of real-time systems (RTSs), software testing is made more difficult by non-deterministic factors such as interaction with the surrounding environment, (pseudo)parallelism, and timing requirements. Furthermore, RTS testing suffers from behaviour-altering perturbation from the instrumentation inserted in the system to keep track of test progress (i.e., probe effects).In our work, we analyse the main test criteria used for traditional software testing in order to see which of these scale to, and assist in, system-level testing of multi-tasking RTSs. We focus on one of these criteria, the all definition-use paths coverage criterion, as it highlights a central aspect of non-deterministic task interaction, and investigate what is specifically required for applying this criterion to testing of multi-tasking RTSs. Further, we examine the possibility of using execution replay for probe effect-free test monitoring. We evaluate this approach in real industrial settings by means of case studies.The contributions of this thesis are twofold: First, the use of structural test criteria in RTS system-level testing is facilitated by two different analysis methods. Second, the probe effect is handled by recording non-deterministic events during run-time, and by using this recording to create a monitorable deterministic replica of the first execution. By these contributions, this thesis shows how the non-determinism of multi-tasking RTSs can be handled during system-level testing.
|
|
| 7. |
- Abbas, Muhammad
(författare)
-
Requirements-Level Reuse Recommendation and Prioritization of Product Line Assets
- 2021
-
Licentiatavhandling (övrigt vetenskapligt/konstnärligt)abstract
- Software systems often target a variety of different market segments. Targeting varying customer requirements requires a product-focused development process. Software Product Line (SPL) engineering is one possible approach based on reuse rationale to aid quick delivery of quality product variants at scale. SPLs reuse common features across derived products while still providing varying configuration options. The common features, in most cases, are realized by reusable assets. In practice, the assets are reused in a clone-and-own manner to reduce the upfront cost of systematic reuse. Besides, the assets are implemented in increments, and requirements prioritization also has to be done. In this context, the manual reuse analysis and prioritization process become impractical when the number of derived products grows. Besides, the manual reuse analysis process is time-consuming and heavily dependent on the experience of engineers.In this licentiate thesis, we study requirements-level reuse recommendation and prioritization for SPL assets in industrial settings. We first identify challenges and opportunities in SPLs where reuse is done in a clone-and-own manner. We then focus on one of the identified challenges: requirements-based SPL assets reuse and provide automated support for identifying reuse opportunities for SPL assets based on requirements. Finally, we provide automated support for requirements prioritization in the presence of dependencies resulting from reuse.
|
|
| 8. |
- Karlsson, Stefan
(författare)
-
Exploring API Behaviours by Example Generation
- 2024
-
Doktorsavhandling (övrigt vetenskapligt/konstnärligt)abstract
- Understanding the behaviour of complex software-intensive systems is a hard task. For developers of such systems, understanding the actual behaviours is critical in order to successfully create, extend, and maintain them.The goal of the work in this thesis is to support explorations of the behaviour of software systems through their APIs. We fulfil this goal by generating examples of behaviours the system exhibits. An example is expressed as a sequence of API operations---with parameters, if required---that conforms to a specific behaviour.Examples of behaviours, such as sequences of operations performed on the system, have been shown to be a good way to further the understanding of software systems for both end users and developers. However, manually creating examples requires effort. In addition, manually created examples only contain what a human can imagine---which might miss important cases, such as unintended behaviours.The main proposed approach in this thesis is to support users in exploring the behaviour of their software system by automatically generating examples of actual behaviour. By only interacting with the system by the exposed API, we assess the behaviours as exposed to an end user of the API. The input to the approach is a set of API operations and schema of operation parameters. Sequences of operations are generated containing these provided operations. The observed responses from executing the generated sequences are used to assess if the API show an example of a sought behaviour. Found examples go through a shrinking process---trying to find a more minimal sequence showing the same behaviour---and are then reported to the user of the approach. The approach is capable of both generating examples of faults in the system and of generating examples of general behaviours. We show evidence of this through multiple evaluations. We have evaluated the fault-finding capabilities by generating examples producing fault-indicating error codes and showing how the configuration of generators affects the interaction with the system. In addition, we evaluate the capability of the approach to generate relevant examples, both in the general API case and in the specific case of REST APIs. By conducting multiple focus group sessions, we conclude that the examples of behaviours produced by the approach indeed aid industry practitioners. The generated examples are deemed relevant for use cases such as testing, documenting, and understanding the behaviour of the system.
|
|
| 9. |
- Karlsson, Stefan
(författare)
-
Towards Augmented Exploratory Testing
- 2021
-
Licentiatavhandling (övrigt vetenskapligt/konstnärligt)abstract
- Software systems have an increasing presence in our society. With our infrastructure, such as food and water supply, controlled by complex software systems, it is essential to keep failures to a minimum. Exploratory testing has shown to be a good method of finding bugs that require more complex interactions, as well as to gain insights into how the system under test behaves. However, tool support in this area is lacking. Supporting exploratory testing with automation tools has the potential of freeing humans to spend their time where it is of the highest yield and also aid in covering the vast state space of testing a complex software system in a meaningful way. To be able to engineer such tools, targeting interaction models of contemporary systems, deeper knowledge is needed of the possibilities and limitations of applying available software testing methods. In this thesis we propose and evaluate several approaches that can be used to automatically perform black-box system-level testing, potentially augmenting humans in exploratory testing. We base these approaches on random-based methods, such as property-based testing, and present the results of evaluations on several industry-grade systems.Our results show that the proposed and evaluated methods automatically can find faults in real-world software systems. In addition to fault finding, our methods can also find insights such as deviations between the specification of the system under test and the actual behavior.The work in this thesis is a first step towards augmenting developers and testers in exploratory testing.
|
|
| 10. |
- Strandberg, Per Erik
(författare)
-
Automated System-Level Software Testing of Industrial Networked Embedded Systems
- 2021
-
Doktorsavhandling (övrigt vetenskapligt/konstnärligt)abstract
- Embedded systems are ubiquitous and play critical roles in management systems for industry and transport. Software failures in these domains may lead to loss of production or even loss of life, so the software in these systems needs to be reliable. Software testing is a standard approach for quality assurance of embedded software, and many software development processes strive for test automation. Out of the many challenges for successful software test automation, this thesis addresses five: (i) understanding how updated software reaches a test environment, how testing is conducted in the test environment, and how test results reach the developers that updated the software in the first place; (ii) selecting which test cases to execute in a test suite given constraints on available time and test systems; (iii) given that the test cases an run on different configurations of connected devices, selecting which hardware to use for each test case to be executed; (iv) analyzing test cases that, when executed over time on evolving software, testware or hardware revisions, appear to randomly fail; and (v) making test results information actionable with test results exploration and visualization.The challenges are tackled in several ways. First, to better understand the flow of information in the embedded systems software development process, interviews at five different companies were conducted. The results show how visualizations and a test results database support decision-making. Results also describe the overall flow of information in software testing: from developers to hardware in the test environment, and back to developers. Second, in order to address the challenges of test selection and hardware selection, automated approaches for testing given resource constraints were implemented and evaluated using industrial data stemming from years of nightly testing. It was shown that these approaches could solve problems such as nightly testing not finishing on time, as well as increasing hardware coverage by varying hardware selection over test iterations. Third, the challenge of intermittently failing tests was addressed with a new metric that can classify test cases as intermittently or consistently failing. Again, by using industry data, factors that lead to intermittent failures were identified, and similarities and differences between root causes for intermittently and consistently failing tests were observed. Finally, in order to better render test results actionable, a tool was implemented for test results exploration and visualization. The implementation was evaluated using a reference group and logging of the tool’s usage. Solution patterns and views of the tool were identified, as well as challenges for implementing such a tool.
|
|
| 11. |
- Tahvili, Sahar
(författare)
-
A Decision Support System for Integration Test Selection
- 2016
-
Licentiatavhandling (övrigt vetenskapligt/konstnärligt)abstract
- Software testing generally suffers from time and budget limitations. Indiscriminately executing all available test cases leads to sub-optimal exploitation of testing resources. Selecting too few test cases for execution on the other hand might leave a large number of faults undiscovered. Test case selection and prioritization techniques can lead to more efficient usage of testing resources and also early detection of faults. Test case selection addresses the problem of selecting a subset of an existing set of test cases, typically by discarding test cases that do not add any value in improving the quality of the software under test. Test case prioritization schedules test cases for execution in an order to increase their effectiveness at achieving some performance goals such as: earlier fault detection, optimal allocation of testing resources and reducing overall testing effort. In practice, prioritized selection of test cases requires the evaluation of different test case criteria, and therefore, this problem can be formulated as a multi-criteria decision making problem. As the number of decision criteria grows, application of a systematic decision making solution becomes a necessity. In this thesis, we propose a tool-supported framework using a decision support system, for prioritizing and selecting integration test cases in embedded system development. The framework provides a complete loop for selecting the best candidate test case for execution based on a finite set of criteria. The results of multiple case studies, done on a train control management subsystem from Bombardier Transportation AB in Sweden, demonstrate how our approach helps to select test cases in a systematic way. This can lead to early detection of faults while respecting various criteria. Also, we have evaluated a customized return on investment metric to quantify the economic benefits in optimizing system integration testing using our framework.
|
|
| 12. |
- Vitucci, Carlo, 1967-
(författare)
-
The role of fault management in the embedded system design
- 2024
-
Licentiatavhandling (övrigt vetenskapligt/konstnärligt)abstract
- In the last decade, the world of telecommunications has seen the value ofservices definitively affirmed and the loss of the connectivity value. This changeof pace in the use of the network (and available hardware resources) has ledto continuous, unlimited growth in data traffic, increased incomes for serviceproviders, and a constant erosion of operators’ incomes for voice and ShortMessage Service (SMS) traffic.The change in mobile service consumption is evident to operators. Themarket today is in the hands of over the top (OTT) media content deliverycompanies (Google, Meta, Netflix, Amazon, etc.), and The fifth generation ofmobile networks (5G), the latest generation of mobile architecture, is nothingother than how operators can invest in system infrastructure to participate in theprosperous service business.With the advent of 5G, the worlds of cloud and telecommunications havefound their meeting point, paving the way for new infrastructures and ser-vices, such as smart cities, industry 4.0, industry 5.0, and Augmented Reality(AR)/Virtual Reality (VR). People, infrastructures, and devices are connected toprovide services that we even struggle to imagine today, but a highly intercon-nected system requires high levels of reliability and resilience.Hardware reliability has increased since the 1990s. However, it is equallycorrect to mention that the introduction of new technologies in the nanometerdomain and the growing complexity of on-chip systems have made fault man-agement critical to guarantee the quality of the service offered to the customerand the sustainability of the network infrastructure.In this thesis, our first contribution is a review of the fault managementimplementation framework for the radio access network domain. Our approachintroduces a holistic vision in fault management where there is increasingly moresignificant attention to the recovery action, the crucial target of the proposedframework. A new contribution underlines the attention toward the recoverytarget: we revisited the taxonomy of faults in mobile systems to enhance theresult of the recovery action, which, in our opinion, must be propagated betweenthe different layers of an embedded system ( hardware, firmware, middleware,and software). The practical adoption of the new framework and the newtaxonomy allowed us to make a unique contribution to the thesis: the proposalof a new algorithm for managing system memory errors, both temporary (soft)and permanent (hard)The holistic vision of error management we introduced in this thesis involveshardware that proactively manages faults. An efficient implementation of faultmanagement is only possible if the hardware design considers error-handlingtechniques and methodologies. Another contribution of this thesis is the def-inition of the fault management requirements for the RAN embedded systemhardware design.Another primary function of the proposed fault management framework isfault prediction. Recognizing error patterns means allowing the system to reactin time, even before the error condition occurs, or identifying the topology of theerror to implement more targeted and, therefore, more efficient recovery actions.The operating temperature is always a critical characteristic of embedded radioaccess network systems. Base stations must be able to work in very differenttemperature conditions. However, the working temperature also directly affectsthe probability of error for the system. In this thesis, we have also contributed interms of a machine-learning algorithm for predicting the working temperature ofbase stations in radio access networks — a first step towards a more sophisticatedimplementation of error prevention and prediction.
|
|
| 13. |
- Wiklund, Kristian, 1972-
(författare)
-
Impediments for Automated Software Test Execution
- 2015
-
Doktorsavhandling (övrigt vetenskapligt/konstnärligt)abstract
- Automated software test execution is a critical part of the modern software development process, where rapid feedback on the product quality is expected. It is of high importance that impediments related to test execution automation are prevented and removed as quickly as possible. An enabling factor for all types of improvement is to understand the nature of what is to be improved. The goal with this thesis is to further the knowledge about common problems encountered by software developers using test execution automation, in order to enable improvement of test execution automation in industrial software development. The research has been performed through industrial case studies and literature reviews. The analysis of the data have primarily been performed using qualitative methods, searching for patterns, themes, and concepts in the data. Our main findings include: (a) a comprehensive list of potential impediments reported in the published body of knowledge on test execution automation, (b) an in-depth analysis of how such impediments may affect the performance of a development team, and (c) a proposal for a qualitative model of interactions between the main groups of phenomena that contribute to the formation of impediments in a test execution automation project. In addition to this, we contribute qualitative and quantitative empirical data from our industrial case studies. Through our results, we find that test execution automation is a commonly under-estimated activity, not only in terms of resources but also in terms of the complexity of the work. There is a clear tendency to perform the work ad hoc, down-prioritize the automation in favor of other activities, and ignore the long-term effects in favor of short-term gains. This is both a technical and a cultural problem that need to be managed by awareness of the problems that may arise, and also have to be solved in the long term through education and information. We conclude by proposing a theoretical model of the socio-technical system that needs to be managed to be successful with test execution automation.
|
|
