| 1. |
- Agirre, Joseba A, et al.
(author)
-
Multidimensional Framework for Characterizing Verification and Validation of Automated Systems
- 2022
-
In: 18<sup>th</sup> European dependable computing conference (EDCC 2022).
-
Conference paper (peer-reviewed)abstract
- Verification and Validation (V&V) of automated systems is becoming more costly and time-consuming because of the increasing size and complexity of these systems. Moreover, V&V of these systems can be hindered if the methods and processes are not properly described, analysed, and selected. It is essential that practitioners use suitable V&V methods and enact adequate V&V processes to confirm that these systems work as intended and in a cost-effective manner. Previous works have created different taxonomies and models considering different aspects of V&V that can be used to classify V&V methods and tools. The aim of this work is to provide a broad, comprehensive and a easy to use framework that addresses characterisation needs, rather than focusing on individual aspects of V&V methods and processes.To this end, in this paper, we present a multi-domain and multi-dimensional framework to characterize and classify V&V methods and tools in a structured way. The framework considers a comprehensive characterization of different relevant aspects of V&V. A web-based repository has been implemented on the basis of the framework, as an example of use, in order to collect information about the application of V&V methods and tools. This way, practitioners and researchers can easily learn about and identify suitable V&V processes.
|
|
| 2. |
- Agirre, J. A., et al.
(author)
-
The VALU3S ECSEL project : Verification and validation of automated systems safety and security
- 2021
-
In: Microprocessors and microsystems. - : Elsevier BV. - 0141-9331 .- 1872-9436. ; 87, s. 104349-
-
Journal article (peer-reviewed)abstract
- Manufacturers of automated systems and their components have been allocating an enormous amount of time and effort in R&D activities, which led to the availability of prototypes demonstrating new capabilities as well as the introduction of such systems to the market within different domains. Manufacturers need to make sure that the systems function in the intended way and according to specifications. This is not a trivial task as system complexity rises dramatically the more integrated and interconnected these systems become with the addition of automated functionality and features to them. This effort translates into an overhead on the V&V (verification and validation) process making it time-consuming and costly. In this paper, we present VALU3S, an ECSEL JU (joint undertaking) project that aims to evaluate the state-of-the-art V&V methods and tools, and design a multi-domain framework to create a clear structure around the components and elements needed to conduct the V&V process. The main expected benefit of the framework is to reduce time and cost needed to verify and validate automated systems with respect to safety, cyber-security, and privacy requirements. This is done through identification and classification of evaluation methods, tools, environments and concepts for V&V of automated systems with respect to the mentioned requirements. VALU3S will provide guidelines to the V&V community including engineers and researchers on how the V&V of automated systems could be improved considering the cost, time and effort of conducting V&V processes. To this end, VALU3S brings together a consortium with partners from 10 different countries, amounting to a mix of 25 industrial partners, 6 leading research institutes, and 10 universities to reach the project goal.
|
|
| 3. |
- Ayatolahi, Fatemeh, 1984, et al.
(author)
-
A study of the impact of single bit-flip and double bit-flip errors on program execution
- 2013
-
In: Lect. Notes Comput. Sci.. - Berlin, Heidelberg : Springer Berlin Heidelberg. - 9783642407925 ; , s. 265-276
-
Conference paper (peer-reviewed)abstract
- This paper presents the results of an extensive experimental study of bit-flip errors in instruction set architecture registers and main memory locations. Comprising more than two million fault injection experiments conducted with thirteen benchmark programs, the study provides insights on whether it is necessary to consider double bit-flip errors in dependability benchmarking experiments. The results show that the proportion of silent data corruptions in the program output, is almost the same for single and double bit errors. In addition, we present detailed statistics about the error sensitivity of different target registers and memory locations, including bit positions within registers and memory words. These show that the error sensitivity varies significantly between different bit positions and registers. An important observation is that injections in certain bit positions always have the same impact regardless of when the error is injected.
|
|
| 4. |
- Barbosa, Raul, et al.
(author)
-
The VALU3S ECSEL Project: Verification and Validation of Automated Systems Safety and Security
- 2020
-
In: 23rd Euromicro Conference on Digital System Design (DSD).
-
Conference paper (peer-reviewed)abstract
- Manufacturers of automated systems and their components have been allocating an enormous amount of time and effort in R&D activities. This effort translates into an overhead on the V&V (verification and validation) process making it timeconsuming and costly. In this paper, we present an ECSEL JU project (VALU3S) that aims to evaluate the state-of-the-art V&V methods and tools, and design a multi-domain framework to create a clear structure around the components and elements needed to conduct the V&V process. The main expected benefit of the framework is to reduce time and cost needed to verify and validate automated systems with respect to safety, cyber-security, and privacy requirements. This is done through identification and classification of evaluation methods, tools, environments and concepts for V&V of automated systems with respect to the mentioned requirements. To this end, VALU3S brings together a consortium with partners from 10 different countries, amounting to a mix of 25 industrial partners, 6 leading research institutes, and 10 universities to reach the project goal.
|
|
| 5. |
- Bauer, Thomas, et al.
(author)
-
Cross-domain Modelling of Verification and Validation Workflows in the Large Scale European Research Project VALU3S
- 2021
-
In: Lecture Notes in Computer Science. - Cham : Springer. ; , s. 386-382
-
Conference paper (peer-reviewed)abstract
- The complexity of systems continues to increase rapidly, especially due to the multi-level integration of subsystems from different domains into cyber-physical systems. This results in special challenges for the efficient verification and validation (V&V) of these systems with regard to their requirements and properties. In order to tackle the new challenges and improve the quality assurance processes, the V&V workflows have to be documented and analyzed. In this paper, a novel approach for the workflow modelling of V&V activities is presented. The generic approach is tailorable to different industrial domains and their specific constraints, V&V methods, and toolchains. The outcomes comprise a dedicated modelling notation (VVML) and tool-support using the modelling framework Enterprise Architect for the efficient documentation and implementation of workflows in the use cases. The solution enables the design of re-usable workflow assets such as V&V activities and artifacts that are exchanged between workflows. This work is part of the large scale European research project VALU3S that deals with the improvement and evaluation of V&V processes in different technical domains, focusing on safety, cybersecurity, and privacy properties.
|
|
| 6. |
- de la Vara, Jose Luis, et al.
(author)
-
A Proposal for the Classification of Methods for Verification and Validation of Safety, Cybersecurity, and Privacy of Automated Systems
- 2021
-
In: 14th International Conference on the Quality of Information and Communications Technology (QUATIC 2021).
-
Conference paper (peer-reviewed)abstract
- As our dependence on automated systems grows, so does the need for guaranteeing their safety, cybersecurity, and privacy (SCP). Dedicated methods for verification and validation (V&V) must be used to this end and it is necessary that the methods and their characteristics can be clearly differentiated. This can be achieved via method classifications. However, we have experienced that existing classifications are not suitable to categorise V&V methods for SCP of automated systems. They do not pay enough attention to the distinguishing characteristics of this system type and of these quality concerns. As a solution, we present a new classification developed in the scope of a large-scale industry-academia project. The classification considers both the method type, e.g., testing, and the concern addressed, e.g., safety. Over 70 people have successfully used the classification on 53 methods. We argue that the classification is a more suitable means to categorise V&V methods for SCP of automated systems and that it can help other researchers and practitioners.
|
|
| 7. |
- Di Leo, D., et al.
(author)
-
On the impact of hardware faults - An investigation of the relationship between workload inputs and failure mode distributions
- 2012
-
In: Lect. Notes Comput. Sci.. - Berlin, Heidelberg : Springer Berlin Heidelberg. - 1611-3349 .- 0302-9743. - 9783642336775 ; , s. 198-209
-
Conference paper (peer-reviewed)abstract
- Technology scaling of integrated circuits is making transistors increasingly sensitive to process variations, wear-out effects and ionizing particles. This may lead to an increasing rate of transient and intermittent errors in future microprocessors. In order to assess the risk such errors pose to safety critical systems, it is essential to investigate how temporary errors in the instruction set architecture (ISA) registers and main memory locations influence the behaviour of executing programs. To this end, we investigate - by means of extensive fault injection experiments - how such errors affect the execution of four target programs. The paper makes three contributions. First, we investigate how the failure modes of the target programs vary for different input sets. Second, we evaluate the error coverage of a software-implemented hardware fault tolerant technique that relies on triple-time redundant execution, majority voting and forward recovery. Third, we propose an approach based on assembly language metrics which can be used to correlate the dynamic fault-free behaviour of a program with its failure mode distribution obtained by fault injection.
|
|
| 8. |
- Farooqui, Ashfaq, et al.
(author)
-
Towards Formal Fault Injection for Safety Assessment of Automated Systems
- 2023
-
In: Fifth International Workshop on Formal Methods for Autonomous Systems.
-
Conference paper (peer-reviewed)abstract
- Reasoning about safety, security, and other dependability attributes of autonomous systems is a challenge that needs to be addressed before the adoption of such systems in day-to-day life. Formal methods is a class of methods that mathematically reason about a system’s behavior. Thus, a correctness proof is sufficient to conclude the system’s dependability. However, these methods are usually applied to abstract models of the system, which might not fully represent the actual system. Fault injection, on the other hand, is a testing method to evaluate the dependability of systems. However, the amount of testing required to evaluate the system is rather large and often a problem. This vision paper introduces formal fault injection, a fusion of these two techniques throughout the development lifecycle to enhance the dependability of autonomous systems. We advocate for a more cohesive approach by identifying five areas of mutual support between formal methods and fault injection. By forging stronger ties between the two fields, we pave the way for developing safe and dependable autonomous systems. This paper delves into the integration’s potential and outlines future research avenues, addressing open challenges along the way.
|
|
| 9. |
- Ferrari, Enrico, et al.
(author)
-
Criteria for the Analysis of Gaps and Limitations of V&V Methods for Safety- and Security-Critical Systems
- 2022
-
Conference paper (peer-reviewed)abstract
- As society increasingly relies on safety- and security- critical systems, the need for confirming their dependability becomes essential. Adequate V&V (verification and validation) methods must be employed, e.g., for system testing. When selecting and using the methods, it is important to analyze their possible gaps and limitations, such as scalability issues. However, and as we have experienced, common, explicitly defined criteria are seldom used for such analyses. This results in analyses that consider different aspects and to a different extent, hindering their comparison and thus the comparison of the V&V methods. As a solution, we present a set of criteria for the analysis of gaps and limitations of V&V methods for safety- and security-critical systems. The criteria have been identified in the scope of the VALU3S project. Sixty-two people from 33 organizations agreed upon the use of nine criteria: functionality, accuracy, scalability, deployment, learning curve, automation, reference environment, cost, and standards. Their use led to more homogeneous and more detailed analyses when compared to similar previous efforts. We argue that the proposed criteria can be helpful to others when having to deal with similar activities.
|
|
| 10. |
- Folkesson, Peter, 1968, et al.
(author)
-
Back-to-Back Fault Injection Testing in Model-Based Development
- 2015
-
In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). - Cham : Springer International Publishing. - 1611-3349 .- 0302-9743. - 9783319242545 - 9783319242552 ; 9337, s. 135-148
-
Conference paper (peer-reviewed)abstract
- Today, embedded systems across industrial domains (e.g., avionics, automotive) are representatives of software-intensive systems with increasing reliance on software and growing complexity. It has become critically important to verify software in a time, resource and cost effective manner. Furthermore, industrial domains are striving to comply with the requirements of relevant safety standards. This paper proposes a novel workflow along with tool support to evaluate robustness of software in model-based development environment, assuming different abstraction levels of representing software. We then show the effectiveness of our technique, on a brake-by-wire application, by performing back-to-back fault injection testing between two different abstraction levels using MODIFI for the Simulink model and GOOFI-2 for the generated code running on the target microcontroller. Our proposed method and tool support facilitates not only verifying software during early phases of the development lifecycle but also fulfilling back-to-back testing requirements of ISO 26262 when using model-based development.
|
|
