| 1. |
- Henriksson, Jens, 1991, et al.
(författare)
-
Towards Structured Evaluation of Deep Neural Network Supervisors
- 2019
-
Ingår i: Proceedings - 2019 IEEE International Conference on Artificial Intelligence Testing, AITest 2019. - New York : Institute of Electrical and Electronics Engineers Inc.. - 9781728104928 ; 1
-
Konferensbidrag (refereegranskat)abstract
- Deep Neural Networks (DNN) have improved the quality of several non-safety related products in the past years. However, before DNNs should be deployed to safety-critical applications, their robustness needs to be systematically analyzed. A common challenge for DNNs occurs when input is dissimilar to the training set, which might lead to high confidence predictions despite proper knowledge of the input. Several previous studies have proposed to complement DNNs with a supervisor that detects when inputs are outside the scope of the network. Most of these supervisors, however, are developed and tested for a selected scenario using a specific performance metric. In this work, we emphasize the need to assess and compare the performance of supervisors in a structured way. We present a framework constituted by four datasets organized in six test cases combined with seven evaluation metrics. The test cases provide varying complexity and include data from publicly available sources as well as a novel dataset consisting of images from simulated driving scenarios. The latter we plan to make publicly available. Our framework can be used to support DNN supervisor evaluation, which in turn could be used to motive development, validation, and deployment of DNNs in safety-critical applications.
|
|
| 2. |
- Schröder, Jan, 1986, et al.
(författare)
-
Challenges from Integration Testing using Interconnected Hardware-in-the-Loop Test Rigs at an Automotive OEM: An Industrial Experience Report
- 2015
-
Ingår i: Proceedings of the First International Workshop on Automotive Software Architecture, WASA '15. - New York, NY, USA : Association for Computing Machinery (ACM). - 9781450334440
-
Konferensbidrag (refereegranskat)abstract
- Developing automotive functions involves complex software to a growing extent while still following a consecutive waterfall-like development process: Integrating and testing software with other software and with hardware components is conducted towards the final phases during the development. For example, ambiguous requirements or unclear semantics in system interfaces show up very late and mostly not before integration testing. In this article, we are reporting about results from conducting interviews with integration and test engineers at a large automotive OEM about today's most resource-intense challenges when dealing with software integration testing tasks at interconnected hardware-in-the-loop test rig environments. Challenges in processes, communication, and implementation were mentioned most often as emerging topics to be tackled over and over again during this phase in a vehicle series development project.
|
|
| 3. |
- Schröder, Jan, 1986, et al.
(författare)
-
Comparing the applicability of complexity measurements for simulink models during integration testing: an industrial case study
- 2015
-
Ingår i: Proceedings of the Second International Workshop on Software Architecture and Metrics, SAM '15. - : IEEE. - 9781467370769
-
Konferensbidrag (refereegranskat)abstract
- Context: Simulink models are used during software integration testing in the automotive domain on hardware in the loop (HIL) rigs. As the amount of software in cars is increasing continuously, the number of Simulink models for control logic and plant models is growing at the same time. Objective: The study aims for investigating the applicability of three approaches for evaluating model complexity in an industrial setting. Additionally, insights on the understanding of maintainability in industry are gathered. Method: Simulink models from two vehicle projects at a German premium car manufacturer are evaluated by applying the following three approaches: Assessing a model's (a) size, (b) structure, and (c) signal routing. Afterwards, an interview study is conducted followed by an on-site workshop in order to validate the findings. Results: The measurements of 65 models resulted in comparable data for the three measurement approaches. Together with the interview studies, conclusions were drawn on how well each approach reflects the experts' opinions. Additionally, it was possible to get insights on maintainability in an industrial setting. Conclusion: By analyzing the results, differences between the three measurement approaches were revealed. The interviews showed that the expert opinion tends to favor the results of the simple size measurements over the measurement including the signal routing.
|
|
| 4. |
- Knauss, Alessia, 1983, et al.
(författare)
-
Paving the roadway for safety of automated vehicles : An empirical study on testing challenges
- 2017
-
Ingår i: IEEE Intelligent Vehicles Symposium, Proceedings. - New York : IEEE. - 9781509048045 ; , s. 1873-1880
-
Konferensbidrag (refereegranskat)abstract
- The technology in the area of automated vehicles is gaining speed and promises many advantages. However, with the recent introduction of conditionally automated driving, we have also seen accidents. Test protocols for both, conditionally automated (e.g., on highways) and automated vehicles do not exist yet and leave researchers and practitioners with different challenges. For instance, current test procedures do not suffice for fully automated vehicles, which are supposed to be completely in charge for the driving task and have no driver as a back up. This paper presents current challenges of testing the functionality and safety of automated vehicles derived from conducting focus groups and interviews with 26 participants from five countries having a background related to testing automotive safety-related topics. We provide an overview of the state-of-practice of testing active safety features as well as challenges that needs to be addressed in the future to ensure safety for automated vehicles. The major challenges identified through the interviews and focus groups, enriched by literature on this topic are related to 1) virtual testing and simulation, 2) safety, reliability, and quality, 3) sensors and sensor models, 4)required scenario complexity and amount of test cases, and 5)handover of responsibility between the driver and the vehicle. © 2017 IEEE.
|
|
| 5. |
- Knauss, Alessia, 1983, et al.
(författare)
-
Software-related challenges of testing automated vehicles
- 2017
-
Ingår i: Proceedings - 2017 IEEE/ACM 39th International Conference on Software Engineering Companion, ICSE-C 2017. - : ACM. - 9781538615898 ; , s. 328-330
-
Konferensbidrag (refereegranskat)abstract
- Automated vehicles are not supposed to fail at any time or in any situations during driving. Thus, vehicle manufactures and proving ground operators are challenged to complement existing test procedures with means to systematically evaluate automated driving. In this paper, we explore software related challenges from testing the safety of automated vehicles. We report on findings from conducting focus groups and interviews including 26 participants (e.g., vehicle manufacturers, suppliers, and researchers) from five countries.
|
|
| 6. |
- Knauss, Alessia, 1983, et al.
(författare)
-
Towards state-of-the-art and future trends in testing of active safety systems
- 2016
-
Ingår i: Proceedings of the 2nd International Workshop on Software Engineering for Smart Cyber-Physical Systems. - New York, NY, USA : ACM. - 9781450341714 ; , s. 36-42
-
Konferensbidrag (refereegranskat)abstract
- Cyber-physical systems integrate computational and physical processes and allow for smart applications like smart transportation, smart buildings, and safety systems. While some applications support us in our daily life, others are safety-critical like vehicular active safety systems. Systematic testing of such critical cyber-physical systems is a crucial, but very challenging task that so far has received little research attention. In this paper, we present an investigation on the state-of-the-art and future trends of testing critical cyber-physical systems on the example of active safety systems for vehicles. We conducted four focus groups with Swedish industrial partners as well as systematically studied proceedings of a recent active safety conference. Our results show that while the main testing processes and scenarios are supported, there is a clear need to enable testing of more complex scenarios in realistic settings, as well as increasing the degree of automating therefor to achieve better repeatability and a more effective test resource usage.
|
|
| 7. |
- Al Mamun, Md Abdullah, 1982, et al.
(författare)
-
Explicating, Understanding and Managing Technical Debt from Self-Driving Miniature Car Projects
- 2014
-
Ingår i: Proceedings of the 30th IEEE International Conference on Software Maintenance and Evolution (ICSME). - Los Alamitos, CA : IEEE. - 9781479967919
-
Konferensbidrag (refereegranskat)abstract
- Technical debt refers to various weaknesses in the design or implementation of a system resulting from trade-offs during software development usually for a quick release. Accumulating such debt over time without reducing it can seriously hamper the reusability and maintainability of the software. The aim of this study is to understand the state of the technical debt in the development of self-driving miniature cars so that proper actions can be planned to reduce the debt to have more reusable and maintainable software. A case study on a selected feature from two self-driving miniature car development projects is performed to assess the technical debt. Additionally, an interview study is conducted involving the developers to relate the findings of the case study with the possible root causes. The result of the study indicates that "the lack of knowledge" is not the primary reason for the accumulation of technical debt from the selected code smells. The root causes are rather in factors like time pressure followed by issues related to software/hardware integration and incomplete refactoring as well as reuse of legacy, third party, or open source code.
|
|
| 8. |
- Zhang, Chi, 1992, et al.
(författare)
-
Cross or Wait? Predicting Pedestrian Interaction Outcomes at Unsignalized Crossings
- 2023
-
Ingår i: IEEE Intelligent Vehicles Symposium, Proceedings. - Anchorage, Alaska, Canada, : IEEE. - 9798350346916 - 9798350346923
-
Konferensbidrag (refereegranskat)abstract
- Predicting pedestrian behavior when interacting with vehicles is one of the most critical challenges in the field of automated driving. Pedestrian crossing behavior is influenced by various interaction factors, including time to arrival, pedestrian waiting time, the presence of zebra crossing, and the properties and personality traits of both pedestrians and drivers. However, these factors have not been fully explored for use in predicting interaction outcomes. In this paper, we use machine learning to predict pedestrian crossing behavior including pedestrian crossing decision, crossing initiation time (CIT), and crossing duration (CD) when interacting with vehicles at unsignalized crossings. Distributed simulator data are utilized for predicting and analyzing the interaction factors. Compared with the logistic regression baseline model, our proposed neural network model improves the prediction accuracy and F1 score by 4.46% and 3.23%, respectively. Our model also reduces the root mean squared error (RMSE) for CIT and CD by 21.56% and 30.14% compared with the linear regression model. Additionally, we have analyzed the importance of interaction factors, and present the results of models using fewer factors. This provides information for model selection in different scenarios with limited input features.
|
|
| 9. |
- Rana, Rakesh, 1983, et al.
(författare)
-
Improving Fault Injection in Automotive Model Based Development using Fault Bypass Modeling
- 2013
-
Ingår i: 2nd Workshop on Software-Based Methods for Robust Embedded Systems (SOBRES '13), Koblenz, Germany, Sep 2013. ; 220, s. 2577-2591
-
Konferensbidrag (övrigt vetenskapligt/konstnärligt)abstract
- Fault injection is widely used for validating dependability of computer systems. These techniques have been traditionally used for testing dependability of the both hardware and software systems. With widespread use of model based development in automotive software development more sophisticated needs arise for using fault injection techniques at the model level, which can yield significant benefits in combination with model-based testing or model mutation. In this paper, we address challenges with injecting faults into behavioral models in terms of analysis of results and propose a framework for distinguishing between correct and incorrect simulation results. The focus is laid on an important challenge encountered when injecting faults in continuous models, i.e. managing system-environment inter-dependencies. We analyze the problem in details and outline an effective approach to deal with this problem.
|
|
| 10. |
- Gil Gómez, Gaspar, et al.
(författare)
-
Improving subjective assessments of vehicle dynamics evaluations by means of computer tablets as digital aid
- 2016
-
Ingår i: Computer software and hardware. - 400 Commonwealth Drive, Warrendale, PA, United States : SAE International.
-
Konferensbidrag (refereegranskat)abstract
- Vehicle dynamics development relies on subjective assessments (SA), which is a resource-intensive procedure requiring both expert drivers and vehicles. Furthermore, development projects becoming shorter and more complex, and increasing demands on quality require higher efficiency.Most research in this area has focused on moving from physical to virtual testing. However, SA remains the central method. Less attention has been given to provide better tools for the SA process itself. One promising approach is to introduce computer-tablets to aid data collection, which has proven to be useful in medical studies. Simple software solutions can eliminate the need to transcribe data and generate more flexible and better maintainable questionnaires. Tablets’ technical features envision promising enhancements of SA, which also enable better correlations to objective metrics, a requirement to improve CAE evaluations.However, it cannot be assumed that a tablet-based solution is feasible in vehicle dynamics SA context. Any distraction might result in low SA quality and safety issues when test-drivers are subjected to high mental workload pushing the vehicles to their performance-limits.In this study, a SA tablet-software for steering feel, handling, and ride was developed and systematically evaluated versus the traditional pen-and-paper method. The results indicate that the new approach is technically feasible in this context, meets more use-cases, and the drivers’ attitude towards it is positive. It increased questionnaire completion and rating resolution while reducing the error rate and transcription time.Although attendees reported that the paper-based approach has advantages from a usability point of view, the benefits of the tablet-based approach enable further process-related advantages.
|
|
