| 1. |
- Helali Moghadam, Mahshid, et al.
(author)
-
An autonomous performance testing framework using self-adaptive fuzzy reinforcement learning
- 2022
-
In: Software quality journal. - : Springer. - 0963-9314 .- 1573-1367. ; , s. 127-159
-
Journal article (peer-reviewed)abstract
- Test automation brings the potential to reduce costs and human effort, but several aspects of software testing remain challenging to automate. One such example is automated performance testing to find performance breaking points. Current approaches to tackle automated generation of performance test cases mainly involve using source code or system model analysis or use-case-based techniques. However, source code and system models might not always be available at testing time. On the other hand, if the optimal performance testing policy for the intended objective in a testing process instead could be learned by the testing system, then test automation without advanced performance models could be possible. Furthermore, the learned policy could later be reused for similar software systems under test, thus leading to higher test efficiency. We propose SaFReL, a self-adaptive fuzzy reinforcement learning-based performance testing framework. SaFReL learns the optimal policy to generate performance test cases through an initial learning phase, then reuses it during a transfer learning phase, while keeping the learning running and updating the policy in the long term. Through multiple experiments in a simulated performance testing setup, we demonstrate that our approach generates the target performance test cases for different programs more efficiently than a typical testing process and performs adaptively without access to source code and performance models. © 2021, The Author(s).
|
|
| 2. |
- Helali Moghadam, Mahshid, et al.
(author)
-
Intelligent Load Testing: Self-adaptive Reinforcement Learning-driven Load Runner
-
Other publication (other academic/artistic)abstract
- Load testing with the aim of generating an effective workload to identify performance issues is a time-consuming and complex challenge, particularly for evolving software systems. Current automated approaches mainly rely on analyzing system models and source code, or modeling of the real system usage. However, that information might not be available all the time or obtaining it might require considerable effort. On the other hand, if the optimal policy for generating the proper test workload resulting in meeting the objectives of the testing can be learned by the testing system, testing would be possible without access to system models or source code. We propose a self-adaptive reinforcement learning-driven load testing agent that learns the optimal policy for test workload generation. The agent can reuse the learned policy in subsequent testing activities such as meeting different types of testing targets. It generates an efficient test workload resulting in meeting the objective of the testing adaptively without access to system models or source code. Our experimental evaluation shows that the proposed self-adaptive intelligent load testing can reach the testing objective with lower cost in terms of the workload size, i.e. the number of generated users, compared to a typical load testing process, and results in productivity benefits in terms of higher efficiency.
|
|
| 3. |
- Helali Moghadam, Mahshid, et al.
(author)
-
Learning-based Response Time Analysis in Real-Time Embedded Systems : A Simulation-based Approach
- 2018
-
In: 1st International Workshop on Software Qualities and their Dependencies, located at the International Conference of Software Engineering (ICSE) 2018 SQUADE'18. - New York, NY, USA : ACM. - 9781450357371 ; , s. 21-24
-
Conference paper (peer-reviewed)abstract
- Response time analysis is an essential task to verify the behavior of real-time systems. Several response time analysis methods have been proposed to address this challenge, particularly for real-time systems with different levels of complexity. Static analysis is a popular approach in this context, but its practical applicability is limited due to the high complexity of the industrial real-time systems, as well as many unpredictable runtime events in these systems. In this work-in-progress paper, we propose a simulationbased response time analysis approach using reinforcement learning to find the execution scenarios leading to the worst-case response time. The approach learns how to provide a practical estimation of the worst-case response time through simulating the program without performing static analysis. Our initial study suggests that the proposed approach could be applicable in the simulation environments of the industrial real-time control systems to provide a practical estimation of the execution scenarios leading to the worst-case response time.
|
|
| 4. |
- Helali Moghadam, Mahshid, et al.
(author)
-
Machine Learning Testing in an ADAS Case Study Using Simulation-Integrated Bio-Inspired Search-Based Testing
- 2022
-
Reports (other academic/artistic)abstract
- This paper presents an extended version of Deeper, a search-based simulation-integrated test solution that generates failure-revealing test scenarios for testing a deep neural network-based lane-keeping system. In the newly proposed version, we utilize a new set of bio-inspired search algorithms, genetic algorithm (GA), (μ+ λ) and (μ,λ) evolution strategies(ES), and particle swarm optimization (PSO), that leverage a quality population seed and domain-specific crossover and mutation operations tailored for the presentation model used for modeling the test scenarios. In order to demonstrate the capabilities of the new test generators within Deeper, we carry out an empirical evaluation and comparison with regard to the results of five participating tools in the cyber-physical systems testing competition at SBST 2021. Our evaluation shows the newly proposed test generators in Deeper not only represent a considerable improvement on the previous version but also prove to be effective and efficient in provoking a considerable number of diverse failure-revealing test scenarios for testing an ML-driven lane-keeping system. They can trigger several failures while promoting test scenario diversity, under a limited test time budget, high target failure severity, and strict speed limit constraints.
|
|
| 5. |
- Helali Moghadam, Mahshid, et al.
(author)
-
Machine learning testing in an ADAS case study using simulation-integrated bio-inspired search-based testing
- 2024
-
In: Journal of Software. - : John Wiley and Sons Ltd. - 2047-7473 .- 2047-7481. ; :5
-
Journal article (peer-reviewed)abstract
- This paper presents an extended version of Deeper, a search-based simulation-integrated test solution that generates failure-revealing test scenarios for testing a deep neural network-based lane-keeping system. In the newly proposed version, we utilize a new set of bio-inspired search algorithms, genetic algorithm (GA), (Formula presented.) and (Formula presented.) evolution strategies (ES), and particle swarm optimization (PSO), that leverage a quality population seed and domain-specific crossover and mutation operations tailored for the presentation model used for modeling the test scenarios. In order to demonstrate the capabilities of the new test generators within Deeper, we carry out an empirical evaluation and comparison with regard to the results of five participating tools in the cyber-physical systems testing competition at SBST 2021. Our evaluation shows the newly proposed test generators in Deeper not only represent a considerable improvement on the previous version but also prove to be effective and efficient in provoking a considerable number of diverse failure-revealing test scenarios for testing an ML-driven lane-keeping system. They can trigger several failures while promoting test scenario diversity, under a limited test time budget, high target failure severity, and strict speed limit constraints.
|
|
| 6. |
- Helali Moghadam, Mahshid, et al.
(author)
-
Performance Testing Using a Smart Reinforcement Learning-Driven Test Agent
- 2021
-
In: 2021 IEEE Congress on Evolutionary Computation (CEC). - 9781728183930 ; , s. 2385-2394
-
Conference paper (peer-reviewed)abstract
- Performance testing with the aim of generating an efficient and effective workload to identify performance issues is challenging. Many of the automated approaches mainly rely on analyzing system models, source code, or extracting the usage pattern of the system during the execution. However, such information and artifacts are not always available. Moreover, all the transactions within a generated workload do not impact the performance of the system the same way, a finely tuned workload could accomplish the test objective in an efficient way. Model-free reinforcement learning is widely used for finding the optimal behavior to accomplish an objective in many decision-making problems without relying on a model of the system. This paper proposes that if the optimal policy (way) for generating test workload to meet a test objective can be learned by a test agent, then efficient test automation would be possible without relying on system models or source code. We present a self-adaptive reinforcement learning-driven load testing agent, RELOAD, that learns the optimal policy for test workload generation and generates an effective workload efficiently to meet the test objective. Once the agent learns the optimal policy, it can reuse the learned policy in subsequent testing activities. Our experiments show that the proposed intelligent load test agent can accomplish the test objective with lower test cost compared to common load testing procedures, and results in higher test efficiency.
|
|
| 7. |
- Helali Moghadam, Mahshid, et al.
(author)
-
Poster : Performance Testing Driven by Reinforcement Learning
- 2020
-
In: 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST). - : Institute of Electrical and Electronics Engineers (IEEE). - 9781728157771 ; , s. 402-405
-
Conference paper (peer-reviewed)abstract
- Performance testing remains a challenge, particularly for complex systems. Different application-, platform- and workload-based factors can influence the performance of software under test. Common approaches for generating platform- and workload-based test conditions are often based on system model or source code analysis, real usage modeling and use-case based design techniques. Nonetheless, creating a detailed performance model is often difficult, and also those artifacts might not be always available during the testing. On the other hand, test automation solutions such as automated test case generation can enable effort and cost reduction with the potential to improve the intended test criteria coverage. Furthermore, if the optimal way (policy) to generate test cases can be learnt by testing system, then the learnt policy can be reused in further testing situations such as testing variants, evolved versions of software, and different testing scenarios. This capability can lead to additional cost and computation time saving in the testing process. In this research, we present an autonomous performance testing framework which uses a model-free reinforcement learning augmented by fuzzy logic and self-adaptive strategies. It is able to learn the optimal policy to generate platform- and workload-based test conditions which result in meeting the intended testing objective without access to system model and source code. The use of fuzzy logic and self-adaptive strategy helps to tackle the issue of uncertainty and improve the accuracy and adaptivity of the proposed learning. Our evaluation experiments show that the proposed autonomous performance testing framework is able to generate the test conditions efficiently and in a way adaptive to varying testing situations.
|
|
| 8. |
|
|
| 9. |
- Zinser, Markus, et al.
(author)
-
Comparison of microscopic and macroscopic approaches to simulating the effects of infrastructure disruptions on railway networks
- 2018
-
In: Proceedings of 7th Transport Research Arena TRA 2018, April 16-19, 2018, Vienna, Austr. - : Zenodo.
-
Conference paper (peer-reviewed)abstract
- The current state-of-the-art in timetable analysis in the presence of disruptions is to use railway microsimulation, which typically yields detailed results on infrastructure or timetable performance. However, micro-simulation is time-consuming and requires a detailed infrastructure model. This paper outlines a macroscopic approach which aims at reducing execution time by restricting the level of detail to high-level relations between significant events. In particular, the effect of disruptions is modelled by sampling delay times from probability distributions obtained from historical data. In this paper, we test whether this approach, given common disruption scenarios, still allows accurate results on delays to be obtained. Two disruption scenarios were simulated in RailSys and with the new method, using limited parameter tuning. In the results, visually similar delay distributions were observed. Although there is some room for improvements in accuracy, the new approach appears promising, and we found no evidence against its suitability in the presence of disruptions.
|
|
| 10. |
- Andersson, Tim, 1989-
(author)
-
Automated Tactile Sensing for Quality Control of Locks Using Machine Learning
- 2024
-
Licentiate thesis (other academic/artistic)abstract
- This thesis delves into the use of Artificial Intelligence (AI) for quality control in manufacturing systems, with a particular focus on anomaly detection through the analysis of torque measurements in rotating mechanical systems. The research specifically examines the effectiveness of torque measurements in quality control of locks, challenging the traditional method that relies on human tactile sense for detecting mechanical anomalies. This conventional approach, while widely used, has been found to yield inconsistent results and poses physical strain on operators. A key aspect of this study involves conducting experiments on locks using torque measurements to identify mechanical anomalies. This method represents a shift from the subjective and physically demanding practice of manually testing each lock. The research aims to demonstrate that an automated, AI-driven approach can offer more consistent and reliable results, thereby improving overall product quality. The development of a machine learning model for this purpose starts with the collection of training data, a process that can be costly and disruptive to normal workflow. Therefore, this thesis also investigates strategies for predicting and minimizing the sample size used for training. Additionally, it addresses the critical need of trustworthiness in AI systems used for final quality control. The research explores how to utilize machine learning models that are not only effective in detecting anomalies but also offers a level of interpretability, avoiding the pitfalls of black box AI models. Overall, this thesis contributes to advancing automated quality control by exploring the state-of-the-art machine learning algorithms for mechanical fault detection, focusing on sample size prediction and minimization and also model interpretability. To the best of the author’s knowledge, it is the first study that evaluates an AI-driven solution for quality control of mechanical locks, marking an innovation in the field.
|
|
