| 1. |
- Flemström, Daniel, 1971-
(författare)
-
Industrial System Level Test Automation
- 2021
-
Doktorsavhandling (övrigt vetenskapligt/konstnärligt)abstract
- Vehicular software systems control and monitor many safety-critical functions, such as automated emergency brakes and anti-spin. These functions are integrated and tested at system level to ensure that the entire system works as intended.Traditionally, these functions or selected combinations of functions are tested in isolation. Although desired, rigorous testing of combinations of functions prior to deployment is seldom possible. One reason is the overwhelming work required to write new test cases for the nearly infinite combinations of functions and driving scenarios. Since software testing already accounts for up to 60% of the software development cost and the execution time in test rigs is expensive, further testing must be achieved by running test cases more in parallel. Moreover, new test cases must be reusable in different driving scenarios to cover more combinations of functions. One solution is to express the test logic in a new way so that it can be executed in parallel, independently of other tests and independent of the input stimuli. This would allow reusing the test logic for different sequences or drive scenarios. Passive testing is one such approach that has not yet been used much for vehicular software due to perceived difficulties, although it is well-established in other domains. One particular problem is how to specify and execute passive test cases for vehicular systems in an, for the test engineer, intuitive and straightforward manner. Further, there is a lack of tool support and knowledge on efficiently applying passive testing in an industrial context. Thus, the overall research goal of this thesis is to propose and evaluate industrially applicable methods and tools for passive testing at the system level of vehicular software systems. The research is based on a series of papers and case studies within the vehicular industry. In contrast to existing specification languages based on formal mathematical expressions, considerable effort was spent creating an intuitive language and an interactive tool, simple but powerful enough, to encourage the industrial application of passive testing. The main contributions include an easy-to-write and easy-to-read description language for passive test cases, an interactive development environment, and knowledge on how to succeed in passive testing in an industrial software development process. The findings of this thesis contribute to making passive testing a viable method for system-level testing and a way of reusing test logic. In the case of a studied train control management system, using passive testing may reuse up to 50% of the test logic for the safety-related requirements and 10% of the non-safety-related requirements.
|
|
| 2. |
- Abbas, Muhammad, et al.
(författare)
-
Automated Reuse Recommendation of Product Line Assets Based on Natural Language Requirements
- 2020
-
Ingår i: Lecture Notes in Computer Science. - Cham : Springer Science and Business Media Deutschland GmbH. - 9783030646936 ; , s. 173-189, s. 173-189, s. 173-189
-
Konferensbidrag (refereegranskat)abstract
- Software product lines (SPLs) are based on reuse rationale to aid quick and quality delivery of complex products at scale. Deriving a new product from a product line requires reuse analysis to avoid redundancy and support a high degree of assets reuse. In this paper, we propose and evaluate automated support for recommending SPL assets that can be reused to realize new customer requirements. Using the existing customer requirements as input, the approach applies natural language processing and clustering to generate reuse recommendations for unseen customer requirements in new projects. The approach is evaluated both quantitatively and qualitatively in the railway industry. Results show that our approach can recommend reuse with 74% accuracy and 57.4% exact match. The evaluation further indicates that the recommendations are relevant to engineers and can support the product derivation and feasibility analysis phase of the projects. The results encourage further study on automated reuse analysis on other levels of abstractions.
|
|
| 3. |
|
|
| 4. |
- Abbas, Muhammad, et al.
(författare)
-
Keywords-based test categorization for Extra-Functional Properties
- 2020
-
Ingår i: 2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). - : IEEE. - 9781728110752 ; , s. 153-156
-
Konferensbidrag (refereegranskat)abstract
- Categorizing existing test specifications can provide insights on coverage of the test suite to extra-functional properties. Manual approaches for test categorization can be time-consuming and prone to error. In this short paper, we propose a semi-automated approach for semantic keywords-based textual test categorization for extra-functional properties. The approach is the first step towards coverage-based test case selection based on extra-functional properties. We report a preliminary evaluation of industrial data for test categorization for safety aspects. Results show that keyword-based approaches can be used to categorize tests for extra-functional properties and can be improved by considering contextual information of keywords.
|
|
| 5. |
- Abbas, Muhammad, et al.
(författare)
-
On the relationship between similar requirements and similar software : A case study in the railway domain
- 2023
-
Ingår i: Requirements Engineering. - : Springer Science and Business Media Deutschland GmbH. - 0947-3602 .- 1432-010X. ; 28, s. 23-47
-
Tidskriftsartikel (refereegranskat)abstract
- Recommender systems for requirements are typically built on the assumption that similar requirements can be used as proxies to retrieve similar software. When a stakeholder proposes a new requirement, natural language processing (NLP)-based similarity metrics can be exploited to retrieve existing requirements, and in turn, identify previously developed code. Several NLP approaches for similarity computation between requirements are available. However, there is little empirical evidence on their effectiveness for code retrieval. This study compares different NLP approaches, from lexical ones to semantic, deep-learning techniques, and correlates the similarity among requirements with the similarity of their associated software. The evaluation is conducted on real-world requirements from two industrial projects from a railway company. Specifically, the most similar pairs of requirements across two industrial projects are automatically identified using six language models. Then, the trace links between requirements and software are used to identify the software pairs associated with each requirements pair. The software similarity between pairs is then automatically computed with JPLag. Finally, the correlation between requirements similarity and software similarity is evaluated to see which language model shows the highest correlation and is thus more appropriate for code retrieval. In addition, we perform a focus group with members of the company to collect qualitative data. Results show a moderately positive correlation between requirements similarity and software similarity, with the pre-trained deep learning-based BERT language model with preprocessing outperforming the other models. Practitioners confirm that requirements similarity is generally regarded as a proxy for software similarity. However, they also highlight that additional aspect comes into play when deciding software reuse, e.g., domain/project knowledge, information coming from test cases, and trace links. Our work is among the first ones to explore the relationship between requirements and software similarity from a quantitative and qualitative standpoint. This can be useful not only in recommender systems but also in other requirements engineering tasks in which similarity computation is relevant, such as tracing and change impact analysis.
|
|
| 6. |
- Abbas, Muhammad, et al.
(författare)
-
Product line adoption in industry : an experience report from the railway domain
- 2020
-
Ingår i: SPLC '20: Proceedings of the 24th ACM Conference on Systems and Software Product Line: Volume A. - New York, NY, USA : Association for Computing Machinery. - 9781450375696 ; , s. 130-141, s. 130-141
-
Konferensbidrag (refereegranskat)abstract
- The software system controlling a train is typically deployed on various hardware architectures and must process various signals across those deployments. The increase of such customization scenarios and the needed adherence of the software to various safety standards in different application domains has led to the adoption of product line engineering within the railway domain. This paper explores the current state-of-practice of software product line development within a team developing industrial embedded software for a train propulsion control system. Evidence is collected using a focus group session with several engineers and through inspection of archival data. We report several benefits and challenges experienced during product line adoption and deployment. Furthermore, we identify and discuss improvement opportunities, focusing mainly on product line evolution and test automation.
|
|
| 7. |
- Abbas, Muhammad
(författare)
-
Requirements-Level Reuse Recommendation and Prioritization of Product Line Assets
- 2021
-
Licentiatavhandling (övrigt vetenskapligt/konstnärligt)abstract
- Software systems often target a variety of different market segments. Targeting varying customer requirements requires a product-focused development process. Software Product Line (SPL) engineering is one possible approach based on reuse rationale to aid quick delivery of quality product variants at scale. SPLs reuse common features across derived products while still providing varying configuration options. The common features, in most cases, are realized by reusable assets. In practice, the assets are reused in a clone-and-own manner to reduce the upfront cost of systematic reuse. Besides, the assets are implemented in increments, and requirements prioritization also has to be done. In this context, the manual reuse analysis and prioritization process become impractical when the number of derived products grows. Besides, the manual reuse analysis process is time-consuming and heavily dependent on the experience of engineers.In this licentiate thesis, we study requirements-level reuse recommendation and prioritization for SPL assets in industrial settings. We first identify challenges and opportunities in SPLs where reuse is done in a clone-and-own manner. We then focus on one of the identified challenges: requirements-based SPL assets reuse and provide automated support for identifying reuse opportunities for SPL assets based on requirements. Finally, we provide automated support for requirements prioritization in the presence of dependencies resulting from reuse.
|
|
| 8. |
- Bilic, Damir, et al.
(författare)
-
Detecting inconsistencies in annotated product line models
- 2020
-
Ingår i: SPLC '20: Proceedings of the 24th ACM Conference on Systems and Software Product Line: Volume A. - New York, NY, USA : Association for Computing Machinery. - 9781450375696 ; , s. 252-262
-
Konferensbidrag (refereegranskat)abstract
- Model-based product line engineering applies the reuse practices from product line engineering with graphical modeling for the specification of software intensive systems. Variability is usually described in separate variability models, while the implementation of the variable systems is specified in system models that use modeling languages such as SysML. Most of the SysML modeling tools with variability support, implement the annotation-based modeling approach. Annotated product line models tend to be error-prone since the modeler implicitly describes every possible variant in a single system model. To identifying variability-related inconsistencies, in this paper, we firstly define restrictions on the use of SysML for annotative modeling in order to avoid situations where resulting instances of the annotated model may contain ambiguous model constructs. Secondly, inter-feature constraints are extracted from the annotated model, based on relations between elements that are annotated with features. By analyzing the constraints, we can identify if the combined variability- and system model can result in incorrect or ambiguous instances. The evaluation of our prototype implementation shows the potential of our approach by identifying inconsistencies in the product line model of our industrial partner which went undetected through several iterations of the model.
|
|
| 9. |
- Bilic, Damir
(författare)
-
Managing Variability in SysML Models of Automotive Systems
- 2020
-
Licentiatavhandling (övrigt vetenskapligt/konstnärligt)abstract
- Organizations developing software-intensive systems inevitably face increasing complexity of developed products, mainly due to rapid advancements in all domains of technology. Many such organizations are considering model-based systems engineering (MBSE) practices to cope with the increasing complexity. The use of models, as a central role during product design, promises to provide benefits such as enhanced communication among system stakeholders, continuous verification, improved design integrity, traceability between requirements and system artifacts and many more. Additionally, products are often built in many variants. That is especially obvious in the automotive domain, where customers have the ability to configure vehicles with hundreds of configuration options. To deal with the variability, a product line engineering approach is often used. It allows the development of a family of similar software-intensive systems that share a common base while being adapted to individual customer requirements.In this thesis, the overall goal is to evaluate and facilitate the combination of the two mentioned approaches, model-based systems engineering and product line engineering, in an industrial environment. To achieve the main thesis goal, it was divided into three separate research goals.The first goal was to identify challenges when applying an annotation-based approach for variant management in SysML models on a use case provided by Volvo Construction Equipment. The aim was to identify and understand challenges when using existing tool support to manage variants in implementation artifacts of existing products. The second research goal was to identify reuse-related challenges in the ``clone-and-own'' based development process of Volvo CE. Moreover, we assess the effects of model-based product line engineering on the identified challenges. Lastly, the third research goal was to develop an approach for consistency checking between variability- and SysML system models. To achieve that, we develop an integrated tool chain for model-based product line engineering that allows the integration of variable artifacts, which are not documented in system models, into the development process. Secondly, we define and develop an approach for consistency checking between variability models that describe the system in terms of features and implementation models that describe how variability is implemented in the product itself, since such support does not exist in current state of the art tools.In conclusion, based on the results from the results of case studies at Volvo CE, it was shown that model-based product line engineering has the potential to improve communication and highlight implications of variability to stakeholders (e.g. to non-technical staff), improve traceability between variability in requirements and variability in design and implementation, improve consistency through constraints between variants and automate repetitive activities.In other words, it shows potential for improving product quality while reducing the development lead time. However, the evaluation and measurement of improvement will be left for future work as measuring the product quality and lead time requires an organizational roll out of model-based product-line engineering.
|
|
| 10. |
- Bilic, Damir, et al.
(författare)
-
Towards a Model-Driven Product Line Engineering Process : An Industrial Case Study
- 2020
-
Ingår i: Proceedings of the 13th Innovations in Software Engineering Conference on Formerly Known as India Software Engineering Conference. - New York, NY, USA : Association for Computing Machinery.
-
Konferensbidrag (refereegranskat)abstract
- Many organizations developing software-intensive systems face challenges with high product complexity and large numbers of variants. In order to effectively maintain and develop these product variants, Product-Line Engineering methods are often considered, while Model-based Systems Engineering practices are commonly utilized to tackle product complexity. In this paper, we report on an industrial case study concerning the ongoing adoption of Product Line Engineering in the Model-based Systems Engineering environment at Volvo Construction Equipment (Volvo CE) in Sweden. In the study, we identify and define a Product Line Engineering process that is aligned with Model-based Systems Engineering activities at the engines control department of Volvo CE. Furthermore, we discuss the implications of the migration from the current development process to a Model-based Product Line Engineering-oriented process. This process, and its implications, are derived by conducting and analyzing interviews with Volvo CE employees, inspecting artifacts and documents, and by means of participant observation. Based on the results of a first system model iteration, we were able to document how Model-based Systems Engineering and variability modeling will affect development activities, work products and stakeholders of the work products.
|
|
| 11. |
- Fattouh, Anas, et al.
(författare)
-
A Real-Time Optimization Model for Production Planning in Quarry Sites
- 2021
-
Ingår i: 2021 IEEE 8th International Conference on Industrial Engineering and Applications, ICIEA 2021. - : Institute of Electrical and Electronics Engineers Inc.. - 9781665428958 ; , s. 32-36
-
Konferensbidrag (refereegranskat)abstract
- A quarry site is used to produce gravel with different granularities based on the customer's needs. The production process consists of different stages that need to be planned in order to achieve high productivity and low cost. During the operation of the quarry site, the operating parameters change, and the site productivity could be affected, therefore, a replanning of the production is needed. This paper presents a real-time optimization model for production planning in a quarry site. The optimization model consists of three modules, a productivity monitoring module, a simulation model, and an optimization algorithm. The productivity monitoring module produces the site productivity based on the current operating parameters of the quarry site. Once the site productivity reaches a specific threshold, the optimization algorithm is run on an updated simulation model to find the optimal values of the site parameters. The proposed real-time optimization model is applied to a real quarry site simulated on a high-fidelity simulator. The case study shows that the proposed optimization model can effectively improve the productivity of the quarry site and hence reduces the wastes and improve environmental sustainability.
|
|
| 12. |
- Karlsson, Stefan, et al.
(författare)
-
Automatic Property-based Testing of GraphQL APIs
- 2021
-
Ingår i: Proceedings - 2021 IEEE/ACM International Conference on Automation of Software Test, AST 2021.
-
Konferensbidrag (refereegranskat)abstract
- In recent years, GraphQL has become a popular way to expose web APIs. With its raise of adoption in industry, the quality of GraphQL APIs must be also assessed, as with any part of a software system, and preferably in an automated manner. However, there is currently a lack of methods to automatically generate tests to exercise GraphQL APIs.In this paper, we propose a method for automatically producing GraphQL queries to test GraphQL APIs. This is achieved using a property-based approach to create a generator for queries based on the GraphQL schema of the system under test.Our evaluation on a real world software system shows that this approach is both effective, in terms of finding real bugs, and efficient, as a complete schema can be covered in seconds. In addition, we evaluate the fault finding capability of the method when seeding known faults. 73% of the seeded faults were found, with room for improvements with regards to domain-specific behavior, a common oracle challenge in automatic test generation.
|
|
| 13. |
- Karlsson, Stefan
(författare)
-
Exploring API Behaviours by Example Generation
- 2024
-
Doktorsavhandling (övrigt vetenskapligt/konstnärligt)abstract
- Understanding the behaviour of complex software-intensive systems is a hard task. For developers of such systems, understanding the actual behaviours is critical in order to successfully create, extend, and maintain them.The goal of the work in this thesis is to support explorations of the behaviour of software systems through their APIs. We fulfil this goal by generating examples of behaviours the system exhibits. An example is expressed as a sequence of API operations---with parameters, if required---that conforms to a specific behaviour.Examples of behaviours, such as sequences of operations performed on the system, have been shown to be a good way to further the understanding of software systems for both end users and developers. However, manually creating examples requires effort. In addition, manually created examples only contain what a human can imagine---which might miss important cases, such as unintended behaviours.The main proposed approach in this thesis is to support users in exploring the behaviour of their software system by automatically generating examples of actual behaviour. By only interacting with the system by the exposed API, we assess the behaviours as exposed to an end user of the API. The input to the approach is a set of API operations and schema of operation parameters. Sequences of operations are generated containing these provided operations. The observed responses from executing the generated sequences are used to assess if the API show an example of a sought behaviour. Found examples go through a shrinking process---trying to find a more minimal sequence showing the same behaviour---and are then reported to the user of the approach. The approach is capable of both generating examples of faults in the system and of generating examples of general behaviours. We show evidence of this through multiple evaluations. We have evaluated the fault-finding capabilities by generating examples producing fault-indicating error codes and showing how the configuration of generators affects the interaction with the system. In addition, we evaluate the capability of the approach to generate relevant examples, both in the general API case and in the specific case of REST APIs. By conducting multiple focus group sessions, we conclude that the examples of behaviours produced by the approach indeed aid industry practitioners. The generated examples are deemed relevant for use cases such as testing, documenting, and understanding the behaviour of the system.
|
|
| 14. |
- Karlsson, Stefan, et al.
(författare)
-
Exploring API behaviours through generated examples
- 2024
-
Ingår i: Software Quality Journal. - : SPRINGER. - 1573-1367 .- 0963-9314. ; 32:2, s. 729-763
-
Tidskriftsartikel (refereegranskat)abstract
- Understanding the behaviour of a system's API can be hard. Giving users access to relevant examples of how an API behaves has been shown to make this easier for them. In addition, such examples can be used to verify expected behaviour or identify unwanted behaviours. Methods for automatically generating examples have existed for a long time. However, state-of-the-art methods rely on either white-box information, such as source code, or on formal specifications of the system behaviour. But what if you do not have access to either? This may be the case, for example, when interacting with a third-party API. In this paper, we present an approach to automatically generate relevant examples of behaviours of an API, without requiring either source code or a formal specification of behaviour. Evaluation on an industry-grade REST API shows that our method can produce small and relevant examples that can help engineers to understand the system under exploration.
|
|
| 15. |
- Karlsson, Stefan, et al.
(författare)
-
Model-based Automated Testing of Mobile Applications : An Industrial Case Study
- 2021
-
Ingår i: Proceedings. - 9781665444569 ; , s. 130-137
-
Konferensbidrag (refereegranskat)abstract
- Automatic testing of mobile applications has been a well-researched area in recent years. However, testing in industry is still a very manual practice, as research results have not been fully transferred and adopted. Considering mobile applications, manual testing has the additional burden of adequate testing posed by a large number of available devices and different configurations, as well as the maintenance and setup of such devices.In this paper, we propose and evaluate the use of a model-based test generation approach, where generated tests are executed on a set of cloud-hosted real mobile devices. By using a model-based approach we generate dynamic, less brittle, and implementation simple test cases. The test execution on multiple real devices with different configurations increase the confidence in the implementation of the system under test. Our evaluation shows that the used approach produces a high coverage of the parts of the application related to user interactions. Nevertheless, the inclusion of external services in test generation is required in order to additionally increase the coverage of the complete application. Furthermore, we present the lessons learned while transferring and implementing this approach in an industrial context and applying it to the real product.
|
|
| 16. |
- Karlsson, Stefan, et al.
(författare)
-
QuickREST : Property-based Test Generation of OpenAPI-Described RESTful APIs
- 2020
-
Ingår i: Proceedings - 2020 IEEE 13th International Conference on Software Testing, Verification and Validation, ICST 2020. - : Institute of Electrical and Electronics Engineers Inc.. - 9781728157771 ; , s. 131-141
-
Konferensbidrag (refereegranskat)abstract
- RESTful APIs are an increasingly common way to expose software systems functionality and it is therefore of high interest to find methods to automatically test and verify such APIs. To lower the barrier for industry adoption, such methods need to be straightforward to use with a low effort. This paper introduces a method to explore the behaviour of a RESTful API. This is done by using automatic property-based tests produced from OpenAPI documents that describe the REST API under test. We describe how this method creates artifacts that can be leveraged both as property-based test generators and as a source of validation for results (i.e., as test oracles). Experimental results, on both industrial and open source services, indicate how this approach is a low effort way of finding real faults. Furthermore, it supports building additional knowledge about the system under test by automatically exposing misalignment of specification and implementation. Since the tests are generated from the OpenAPI document this method automatically evolves test cases as the REST API evolves.
|
|
| 17. |
- Karlsson, Stefan
(författare)
-
Towards Augmented Exploratory Testing
- 2021
-
Licentiatavhandling (övrigt vetenskapligt/konstnärligt)abstract
- Software systems have an increasing presence in our society. With our infrastructure, such as food and water supply, controlled by complex software systems, it is essential to keep failures to a minimum. Exploratory testing has shown to be a good method of finding bugs that require more complex interactions, as well as to gain insights into how the system under test behaves. However, tool support in this area is lacking. Supporting exploratory testing with automation tools has the potential of freeing humans to spend their time where it is of the highest yield and also aid in covering the vast state space of testing a complex software system in a meaningful way. To be able to engineer such tools, targeting interaction models of contemporary systems, deeper knowledge is needed of the possibilities and limitations of applying available software testing methods. In this thesis we propose and evaluate several approaches that can be used to automatically perform black-box system-level testing, potentially augmenting humans in exploratory testing. We base these approaches on random-based methods, such as property-based testing, and present the results of evaluations on several industry-grade systems.Our results show that the proposed and evaluated methods automatically can find faults in real-world software systems. In addition to fault finding, our methods can also find insights such as deviations between the specification of the system under test and the actual behavior.The work in this thesis is a first step towards augmenting developers and testers in exploratory testing.
|
|
| 18. |
- Strandberg, Per Erik
(författare)
-
Automated System-Level Software Testing of Industrial Networked Embedded Systems
- 2021
-
Doktorsavhandling (övrigt vetenskapligt/konstnärligt)abstract
- Embedded systems are ubiquitous and play critical roles in management systems for industry and transport. Software failures in these domains may lead to loss of production or even loss of life, so the software in these systems needs to be reliable. Software testing is a standard approach for quality assurance of embedded software, and many software development processes strive for test automation. Out of the many challenges for successful software test automation, this thesis addresses five: (i) understanding how updated software reaches a test environment, how testing is conducted in the test environment, and how test results reach the developers that updated the software in the first place; (ii) selecting which test cases to execute in a test suite given constraints on available time and test systems; (iii) given that the test cases an run on different configurations of connected devices, selecting which hardware to use for each test case to be executed; (iv) analyzing test cases that, when executed over time on evolving software, testware or hardware revisions, appear to randomly fail; and (v) making test results information actionable with test results exploration and visualization.The challenges are tackled in several ways. First, to better understand the flow of information in the embedded systems software development process, interviews at five different companies were conducted. The results show how visualizations and a test results database support decision-making. Results also describe the overall flow of information in software testing: from developers to hardware in the test environment, and back to developers. Second, in order to address the challenges of test selection and hardware selection, automated approaches for testing given resource constraints were implemented and evaluated using industrial data stemming from years of nightly testing. It was shown that these approaches could solve problems such as nightly testing not finishing on time, as well as increasing hardware coverage by varying hardware selection over test iterations. Third, the challenge of intermittently failing tests was addressed with a new metric that can classify test cases as intermittently or consistently failing. Again, by using industry data, factors that lead to intermittent failures were identified, and similarities and differences between root causes for intermittently and consistently failing tests were observed. Finally, in order to better render test results actionable, a tool was implemented for test results exploration and visualization. The implementation was evaluated using a reference group and logging of the tool’s usage. Solution patterns and views of the tool were identified, as well as challenges for implementing such a tool.
|
|
| 19. |
- Strandberg, Per Erik, et al.
(författare)
-
Intermittently Failing Tests in the Embedded Systems Domain
- 2020
-
Ingår i: International Symposium on Software Testing and Analysis ISSTA'20. - New York, NY, USA : ACM. ; , s. 337-348
-
Konferensbidrag (refereegranskat)abstract
- Software testing is sometimes plagued with intermittently failing tests and finding the root causes of such failing tests is often difficult. This problem has been widely studied at the unit testing level for open source software, but there has been far less investigation at the system test level, particularly the testing of industrial embedded systems. This paper describes our investigation of the root causes of intermittently failing tests in the embedded systems domain, with the goal of better understanding, explaining and categorizing the underlying faults. The subject of our investigation is a currently-running industrial embedded system, along with the system level testing that was performed. We devised and used a novel metric for classifying test cases as intermittent. From more than a half million test verdicts, we identified intermittently and consistently failing tests, and identified their root causes using multiple sources. We found that about 1-3% of all test cases were intermittently failing. From analysis of the case study results and related work, we identified nine factors associated with test case intermittence. We found that a fix for a consistently failing test typically removed a larger number of failures detected by other tests than a fix for an intermittent test. We also found that more effort was usually needed to identify fixes for intermittent tests than for consistent tests. An overlap between root causes leading to intermittent and consistent tests was identified. Many root causes of intermittence are the same in industrial embedded systems and open source software. However, when comparing unit testing to system level testing, especially for embedded systems, we observed that the test environment itself is often the cause of intermittence.
|
|
| 20. |
- Strandberg, Per Erik, et al.
(författare)
-
Software test results exploration and visualization with continuous integration and nightly testing
- 2022
-
Ingår i: International Journal on Software Tools for Technology Transfer. - : SPRINGER HEIDELBERG. - 1433-2779 .- 1433-2787. ; 24:2, s. 261-285
-
Tidskriftsartikel (refereegranskat)abstract
- Software testing is key for quality assurance of embedded systems. However, with increased development pace, the amount of test results data risks growing to a level where exploration and visualization of the results are unmanageable. This paper covers a tool, Tim, implemented at a company developing embedded systems, where software development occurs in parallel branches and nightly testing is partitioned over software branches, test systems and test cases. Tim aims to replace a previous solution with problems of scalability, requirements and technological flora. Tim was implemented with a reference group over several months. For validation, data were collected both from reference group meetings and logs from the usage of the tool. Data were analyzed quantitatively and qualitatively. The main contributions from the study include the implementation of eight views for test results exploration and visualization, the identification of four solutions patterns for these views (filtering, aggregation, previews and comparisons), as well as six challenges frequently discussed at reference group meetings (expectations, anomalies, navigation, integrations, hardware details and plots). Results are put in perspective with related work and future work is proposed, e.g., enhanced anomaly detection and integrations with more systems such as risk management, source code and requirements repositories.
|
|
| 21. |
- Thörn, Jonathan, et al.
(författare)
-
Quality assuring the quality assurance tool : Applying safety-critical concepts to test framework development
- 2022
-
Ingår i: PeerJ Computer Science. - : PeerJ. - 2376-5992. ; 8, s. e1131-e1131
-
Tidskriftsartikel (refereegranskat)abstract
- The quality of embedded systems is demonstrated by the performed tests. The quality of such tests is often dependent on the quality of one or more testing tools, especially in automated testing. Test automation is also central to the success of agile development. It is thus critical to ensure the quality of testing tools. This work explores how industries with agile processes can learn from safety-critical system development with regards to the quality assurance of the test framework development. Safety-critical systems typically need adherence to safety standards that often suggests substantial upfront documentation, plans and a long-term perspective on several development aspects. In contrast, agile approaches focus on quick adaptation, evolving software and incremental deliveries. This article identifies several approaches of quality assurance of software development tools in functional safety development and agile development. The extracted approaches are further analyzed and processed into candidate solutions, i.e., principles and practices for the test framework quality assurance applicable in an industrial context. An industrial focus group with experienced practitioners further validated the candidate solutions through moderated group discussions. The two main contributions from this study are: (i) 48 approaches and 25 derived candidate solutions for test framework quality assurance in four categories (development, analysis, run-time measures, and validation and verification) with related insights, e.g., a test framework should be perceived as a tool-chain and not a single tool, (ii) the perceived value of the candidate solutions in industry as collected from the focus group.
|
|
| 22. |
- Vitucci, Carlo, et al.
(författare)
-
A Reliability-oriented Faults Taxonomy and a Recovery-oriented Methodological Approach for Systems Resilience
- 2022
-
Ingår i: Proceedings - 2022 IEEE 46th Annual Computers, Software, and Applications Conference, COMPSAC 2022. - : IEEE. - 9781665488105 ; , s. 48-55
-
Konferensbidrag (refereegranskat)abstract
- Fault management is an important function that impacts the design of any digital system, from the simple kiosk in a shop to a complex 6G network. It is common to classify fault conditions into different taxonomies using terms like fault or error. Fault taxonomies are often suitable for managing fault detection, fault reporting, and fault localization but often neglect to support all different functions required by a fault management process. A correctly implemented fault management process must be able to distinguish between defects and faults, decide upon ap-propriate actions to recover the system to an ideal state, and avoid an error condition. Fault management is a multi-disciplinary process where recovery actions are deployed promptly by com-bined hardware, firmware, and software orchestration. The importance of fault management processes significantly increases with modern nanometer technologies, which suffer the risk of so-called soft errors, a corruption of a bit cells that can happen due to spurious disturbance, like cosmic radiation. Modern fault management implementations must support recovery actions for soft errors to ensure a steady system. This paper describes an extended fault classification model that emphasizes fault management and recovery actions. We aim to show how the reliability-based fault taxonomy definition is more suitable for the overall fault management process.
|
|
| 23. |
- Vitucci, Carlo, 1967-, et al.
(författare)
-
Fault Management Framework and Multi-layer Recovery Methodology for Resilient System
- 2022
-
Ingår i: 2022 6th International Conference on System Reliability and Safety, ICSRS 2022. - : Institute of Electrical and Electronics Engineers Inc.. - 9781665470926 ; , s. 32-39
-
Konferensbidrag (refereegranskat)abstract
- Fault management is a key function to guarantee the quality of the service. Research has done a lot to improve fault supervision, and investigation is ongoing in fault prediction, thanks to the potentials of artificial intelligence and machine learning. In this study, we propose a fault management framework that puts an emphasis on fault recovery: a framework developed on multi-layer function and a fault recovery methodology distributed over several technological layers. The basic principle of our proposal is that the system's complexity exposes it to a higher probability of temporary error. Newfound attention to the fault recovery phase is the key to keeping the service's quality high and saving maintenance costs by decreasing the return rate.
|
|
| 24. |
- Vitucci, Carlo, 1967-, et al.
(författare)
-
Fault Management Impacts on the Networking Systems Hardware Design
- 2023
-
Ingår i: IECON Proceedings (Industrial Electronics Conference). - : IEEE Computer Society. - 9798350331820
-
Konferensbidrag (refereegranskat)abstract
- Processing capacity distribution has become widespread in the fog computing era. End-user services have multiplied, from consumer products to Industry 5.0. In this scenario, the services must have a very high-reliability level. But in a system with such displacement of hardware, the reliability of the service necessarily passes through the hardware design. Devices shall have a high quality, but they shall also efficiently support fault management. Hardware design must take into account all fault management functions and participate in creating a fault management policy to ensure that the ultimate goal of fault management is fulfilled, namely to increase a system's reliability. Efficiently and sustainably, both in the system's performance and the product's cost. This paper analyzes the hardware design techniques that efficiently contribute to the realization of fault management and, consequently, guarantee a high level of reliability and availability for the services offered to the end customer. We describe hardware requirements and how they affect the choice of devices in the hardware design of networking systems.
|
|
| 25. |
- Vitucci, Carlo, 1967-
(författare)
-
The role of fault management in the embedded system design
- 2024
-
Licentiatavhandling (övrigt vetenskapligt/konstnärligt)abstract
- In the last decade, the world of telecommunications has seen the value ofservices definitively affirmed and the loss of the connectivity value. This changeof pace in the use of the network (and available hardware resources) has ledto continuous, unlimited growth in data traffic, increased incomes for serviceproviders, and a constant erosion of operators’ incomes for voice and ShortMessage Service (SMS) traffic.The change in mobile service consumption is evident to operators. Themarket today is in the hands of over the top (OTT) media content deliverycompanies (Google, Meta, Netflix, Amazon, etc.), and The fifth generation ofmobile networks (5G), the latest generation of mobile architecture, is nothingother than how operators can invest in system infrastructure to participate in theprosperous service business.With the advent of 5G, the worlds of cloud and telecommunications havefound their meeting point, paving the way for new infrastructures and ser-vices, such as smart cities, industry 4.0, industry 5.0, and Augmented Reality(AR)/Virtual Reality (VR). People, infrastructures, and devices are connected toprovide services that we even struggle to imagine today, but a highly intercon-nected system requires high levels of reliability and resilience.Hardware reliability has increased since the 1990s. However, it is equallycorrect to mention that the introduction of new technologies in the nanometerdomain and the growing complexity of on-chip systems have made fault man-agement critical to guarantee the quality of the service offered to the customerand the sustainability of the network infrastructure.In this thesis, our first contribution is a review of the fault managementimplementation framework for the radio access network domain. Our approachintroduces a holistic vision in fault management where there is increasingly moresignificant attention to the recovery action, the crucial target of the proposedframework. A new contribution underlines the attention toward the recoverytarget: we revisited the taxonomy of faults in mobile systems to enhance theresult of the recovery action, which, in our opinion, must be propagated betweenthe different layers of an embedded system ( hardware, firmware, middleware,and software). The practical adoption of the new framework and the newtaxonomy allowed us to make a unique contribution to the thesis: the proposalof a new algorithm for managing system memory errors, both temporary (soft)and permanent (hard)The holistic vision of error management we introduced in this thesis involveshardware that proactively manages faults. An efficient implementation of faultmanagement is only possible if the hardware design considers error-handlingtechniques and methodologies. Another contribution of this thesis is the def-inition of the fault management requirements for the RAN embedded systemhardware design.Another primary function of the proposed fault management framework isfault prediction. Recognizing error patterns means allowing the system to reactin time, even before the error condition occurs, or identifying the topology of theerror to implement more targeted and, therefore, more efficient recovery actions.The operating temperature is always a critical characteristic of embedded radioaccess network systems. Base stations must be able to work in very differenttemperature conditions. However, the working temperature also directly affectsthe probability of error for the system. In this thesis, we have also contributed interms of a machine-learning algorithm for predicting the working temperature ofbase stations in radio access networks — a first step towards a more sophisticatedimplementation of error prevention and prediction.
|
|
