| 1. |
- Helali Moghadam, Mahshid, et al.
(författare)
-
An autonomous performance testing framework using self-adaptive fuzzy reinforcement learning
- 2022
-
Ingår i: Software quality journal. - : Springer. - 0963-9314 .- 1573-1367. ; , s. 127-159
-
Tidskriftsartikel (refereegranskat)abstract
- Test automation brings the potential to reduce costs and human effort, but several aspects of software testing remain challenging to automate. One such example is automated performance testing to find performance breaking points. Current approaches to tackle automated generation of performance test cases mainly involve using source code or system model analysis or use-case-based techniques. However, source code and system models might not always be available at testing time. On the other hand, if the optimal performance testing policy for the intended objective in a testing process instead could be learned by the testing system, then test automation without advanced performance models could be possible. Furthermore, the learned policy could later be reused for similar software systems under test, thus leading to higher test efficiency. We propose SaFReL, a self-adaptive fuzzy reinforcement learning-based performance testing framework. SaFReL learns the optimal policy to generate performance test cases through an initial learning phase, then reuses it during a transfer learning phase, while keeping the learning running and updating the policy in the long term. Through multiple experiments in a simulated performance testing setup, we demonstrate that our approach generates the target performance test cases for different programs more efficiently than a typical testing process and performs adaptively without access to source code and performance models. © 2021, The Author(s).
|
|
| 2. |
- Helali Moghadam, Mahshid
(författare)
-
Intelligence-Driven Software Performance Assurance
- 2022
-
Doktorsavhandling (övrigt vetenskapligt/konstnärligt)abstract
- Software performance assurance is of great importance for the success of software products, which are nowadays involved in many parts of our life. Performance evaluation approaches such as performance modeling, testing, as well as runtime performance control methods, all can contribute to the realization of software performance assurance. Many of the common approaches to tackle challenges in this area involve relying on performance models or using system models and source code. Although modeling provides a deep insight into the system behavior, developing a detailed model is challenging. Furthermore, software artifacts such as models and source code might not be readily available at all times in the development lifecycle. This thesis focuses on leveraging the potential of machine learning (ML) and evolutionary search-based techniques to provide viable solutions for addressing the challenges in different aspects of software performance assurance efficiently and effectively.In this thesis, we first investigate the capabilities of model-free reinforcement learning to address the objectives in robustness testing problems. We develop two self-adaptive reinforcement learning-driven test agents called SaFReL and RELOAD. They generate effective platform-based test scenarios and test workloads, respectively. The output scenarios and workloads help testers and software engineers meet their objectives efficiently without relying on models or source code. SaFReL and RELOAD learn the optimal policies (ways) to meet the test objectives and can reuse the learned policies adaptively in other testing settings. Policy reuse can lead to higher test efficiency and cost savings, for example, when testing similar test objectives or software systems with comparable performance sensitivity.Next, we leverage the potential of evolutionary computation algorithms, i.e., genetic algorithms, evolution strategies, and particle swarm optimization, to generate failure-revealing test scenarios for robustness testing of AI systems. In this part, we choose autonomous driving systems as a prevailing example of contemporary AI systems. We study the efficacy of the proposed evolutionary search-based test generation techniques and evaluate primarily to what extent they can trigger failures. Moreover, we investigate the diversity of those failures and compare them to existing baseline solutions. Finally, we again use the potential of model-free reinforcement learning to develop adaptive ML-driven runtime performance control approaches. We present a response time preservation method for a sample type of industrial applications and a resource allocation technique for dynamic workloads in a data grid application. The proposed ML-driven techniques learn how to adjust the tunable parameters and resource configuration at runtime to keep the performance continually compliant with the requirements and to further optimize the runtime performance. We evaluate the efficacy of the approaches and show how effectively they can improve the performance and keep the performance requirements satisfied under varying conditions such as dynamic workloads and the occurrence of runtime events that lead to substantial response time deviations.
|
|
| 3. |
- Helali Moghadam, Mahshid
(författare)
-
Machine Learning-Assisted Performance Assurance
- 2020
-
Licentiatavhandling (övrigt vetenskapligt/konstnärligt)abstract
- With the growing involvement of software systems in our life, assurance of performance, as an important quality characteristic, rises to prominence for the success of software products. Performance testing, preservation, and improvement all contribute to the realization of performance assurance. Common approaches to tackle challenges in testing, preservation, and improvement of performance mainly involve techniques relying on performance models or using system models or source code. Although modeling provides a deep insight into the system behavior, drawing a well-detailed model is challenging. On the other hand, those artifacts such as models and source code might not be available all the time. These issues are the motivations for using model-free machine learning techniques such as model-free reinforcement learning to address the related challenges in performance assurance.Reinforcement learning implies that if the optimal policy (way) for achieving the intended objective in a performance assurance process could instead be learnt by the acting system (e.g., the tester system), then the intended objective could be accomplished without advanced performance models. Furthermore, the learnt policy could later be reused in similar situations, which leads to efficiency improvement by saving computation time while reducing the dependency on the models and source code.In this thesis, our research goal is to develop adaptive and efficient performance assurance techniques meeting the intended objectives without access to models and source code. We propose three model-free learning-based approaches to tackle the challenges; efficient generation of performance test cases, runtime performance (response time) preservation, and performance improvement in terms of makespan (completion time) reduction. We demonstrate the efficiency and adaptivity of our approaches based on experimental evaluations conducted on the research prototype tools, i.e. simulation environments that we developed or tailored for our problems, in different application areas.
|
|
| 4. |
- Helali Moghadam, Mahshid, et al.
(författare)
-
Machine Learning Testing in an ADAS Case Study Using Simulation-Integrated Bio-Inspired Search-Based Testing
- 2022
-
Rapport (övrigt vetenskapligt/konstnärligt)abstract
- This paper presents an extended version of Deeper, a search-based simulation-integrated test solution that generates failure-revealing test scenarios for testing a deep neural network-based lane-keeping system. In the newly proposed version, we utilize a new set of bio-inspired search algorithms, genetic algorithm (GA), (μ+ λ) and (μ,λ) evolution strategies(ES), and particle swarm optimization (PSO), that leverage a quality population seed and domain-specific crossover and mutation operations tailored for the presentation model used for modeling the test scenarios. In order to demonstrate the capabilities of the new test generators within Deeper, we carry out an empirical evaluation and comparison with regard to the results of five participating tools in the cyber-physical systems testing competition at SBST 2021. Our evaluation shows the newly proposed test generators in Deeper not only represent a considerable improvement on the previous version but also prove to be effective and efficient in provoking a considerable number of diverse failure-revealing test scenarios for testing an ML-driven lane-keeping system. They can trigger several failures while promoting test scenario diversity, under a limited test time budget, high target failure severity, and strict speed limit constraints.
|
|
| 5. |
- Helali Moghadam, Mahshid, et al.
(författare)
-
Machine learning testing in an ADAS case study using simulation-integrated bio-inspired search-based testing
- 2024
-
Ingår i: Journal of Software. - : John Wiley and Sons Ltd. - 2047-7473 .- 2047-7481. ; :5
-
Tidskriftsartikel (refereegranskat)abstract
- This paper presents an extended version of Deeper, a search-based simulation-integrated test solution that generates failure-revealing test scenarios for testing a deep neural network-based lane-keeping system. In the newly proposed version, we utilize a new set of bio-inspired search algorithms, genetic algorithm (GA), (Formula presented.) and (Formula presented.) evolution strategies (ES), and particle swarm optimization (PSO), that leverage a quality population seed and domain-specific crossover and mutation operations tailored for the presentation model used for modeling the test scenarios. In order to demonstrate the capabilities of the new test generators within Deeper, we carry out an empirical evaluation and comparison with regard to the results of five participating tools in the cyber-physical systems testing competition at SBST 2021. Our evaluation shows the newly proposed test generators in Deeper not only represent a considerable improvement on the previous version but also prove to be effective and efficient in provoking a considerable number of diverse failure-revealing test scenarios for testing an ML-driven lane-keeping system. They can trigger several failures while promoting test scenario diversity, under a limited test time budget, high target failure severity, and strict speed limit constraints.
|
|
| 6. |
- Helali Moghadam, Mahshid, et al.
(författare)
-
Performance Testing Using a Smart Reinforcement Learning-Driven Test Agent
- 2021
-
Ingår i: 2021 IEEE Congress on Evolutionary Computation (CEC). - 9781728183930 ; , s. 2385-2394
-
Konferensbidrag (refereegranskat)abstract
- Performance testing with the aim of generating an efficient and effective workload to identify performance issues is challenging. Many of the automated approaches mainly rely on analyzing system models, source code, or extracting the usage pattern of the system during the execution. However, such information and artifacts are not always available. Moreover, all the transactions within a generated workload do not impact the performance of the system the same way, a finely tuned workload could accomplish the test objective in an efficient way. Model-free reinforcement learning is widely used for finding the optimal behavior to accomplish an objective in many decision-making problems without relying on a model of the system. This paper proposes that if the optimal policy (way) for generating test workload to meet a test objective can be learned by a test agent, then efficient test automation would be possible without relying on system models or source code. We present a self-adaptive reinforcement learning-driven load testing agent, RELOAD, that learns the optimal policy for test workload generation and generates an effective workload efficiently to meet the test objective. Once the agent learns the optimal policy, it can reuse the learned policy in subsequent testing activities. Our experiments show that the proposed intelligent load test agent can accomplish the test objective with lower test cost compared to common load testing procedures, and results in higher test efficiency.
|
|
| 7. |
- Helali Moghadam, Mahshid, et al.
(författare)
-
Poster : Performance Testing Driven by Reinforcement Learning
- 2020
-
Ingår i: 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST). - : Institute of Electrical and Electronics Engineers (IEEE). - 9781728157771 ; , s. 402-405
-
Konferensbidrag (refereegranskat)abstract
- Performance testing remains a challenge, particularly for complex systems. Different application-, platform- and workload-based factors can influence the performance of software under test. Common approaches for generating platform- and workload-based test conditions are often based on system model or source code analysis, real usage modeling and use-case based design techniques. Nonetheless, creating a detailed performance model is often difficult, and also those artifacts might not be always available during the testing. On the other hand, test automation solutions such as automated test case generation can enable effort and cost reduction with the potential to improve the intended test criteria coverage. Furthermore, if the optimal way (policy) to generate test cases can be learnt by testing system, then the learnt policy can be reused in further testing situations such as testing variants, evolved versions of software, and different testing scenarios. This capability can lead to additional cost and computation time saving in the testing process. In this research, we present an autonomous performance testing framework which uses a model-free reinforcement learning augmented by fuzzy logic and self-adaptive strategies. It is able to learn the optimal policy to generate platform- and workload-based test conditions which result in meeting the intended testing objective without access to system model and source code. The use of fuzzy logic and self-adaptive strategy helps to tackle the issue of uncertainty and improve the accuracy and adaptivity of the proposed learning. Our evaluation experiments show that the proposed autonomous performance testing framework is able to generate the test conditions efficiently and in a way adaptive to varying testing situations.
|
|
| 8. |
- Andersson, Tim, 1989-
(författare)
-
Automated Tactile Sensing for Quality Control of Locks Using Machine Learning
- 2024
-
Licentiatavhandling (övrigt vetenskapligt/konstnärligt)abstract
- This thesis delves into the use of Artificial Intelligence (AI) for quality control in manufacturing systems, with a particular focus on anomaly detection through the analysis of torque measurements in rotating mechanical systems. The research specifically examines the effectiveness of torque measurements in quality control of locks, challenging the traditional method that relies on human tactile sense for detecting mechanical anomalies. This conventional approach, while widely used, has been found to yield inconsistent results and poses physical strain on operators. A key aspect of this study involves conducting experiments on locks using torque measurements to identify mechanical anomalies. This method represents a shift from the subjective and physically demanding practice of manually testing each lock. The research aims to demonstrate that an automated, AI-driven approach can offer more consistent and reliable results, thereby improving overall product quality. The development of a machine learning model for this purpose starts with the collection of training data, a process that can be costly and disruptive to normal workflow. Therefore, this thesis also investigates strategies for predicting and minimizing the sample size used for training. Additionally, it addresses the critical need of trustworthiness in AI systems used for final quality control. The research explores how to utilize machine learning models that are not only effective in detecting anomalies but also offers a level of interpretability, avoiding the pitfalls of black box AI models. Overall, this thesis contributes to advancing automated quality control by exploring the state-of-the-art machine learning algorithms for mechanical fault detection, focusing on sample size prediction and minimization and also model interpretability. To the best of the author’s knowledge, it is the first study that evaluates an AI-driven solution for quality control of mechanical locks, marking an innovation in the field.
|
|
| 9. |
- Andersson, Tim, et al.
(författare)
-
Comparison of Machine Learning’s- and Humans’- Ability to Consistently Classify Anomalies in Cylinder Locks
- 2022
-
Ingår i: IFIP Advances in Information and Communication Technology. - Cham : Springer Science and Business Media Deutschland GmbH. - 9783031164064 ; , s. 27-34, s. 27-34
-
Konferensbidrag (refereegranskat)abstract
- Historically, cylinder locks’ quality has been tested manually by human operators after full assembly. The frequency and the characteristics of the testing procedure for these locks wear the operators’ wrists and lead to varying results of the quality control. The consistency in the quality control is an important factor for the expected lifetime of the locks which is why the industry seeks an automated solution. This study evaluates how consistently the operators can classify a collection of locks, using their tactile sense, compared to a more objective approach, using torque measurements and Machine Learning (ML). These locks were deliberately chosen because they are prone to get inconsistent classifications, which means that there is no ground truth of how to classify them. The ML algorithms were therefore evaluated with two different labeling approaches, one based on the results from the operators, using their tactile sense to classify into ‘working’ or ‘faulty’ locks, and a second approach by letting an unsupervised learner create two clusters of the data which were then labeled by an expert using visual inspection of the torque diagrams. The results show that an ML-solution, trained with the second approach, can classify mechanical anomalies, based on torque data, more consistently compared to operators, using their tactile sense. These findings are a crucial milestone for the further development of a fully automated test procedure that has the potential to increase the reliability of the quality control and remove an injury-prone task from the operators.
|
|
| 10. |
- Andersson, Tim, et al.
(författare)
-
Interpretable ML model for quality control of locks using counterfactual explanations
- 2024
-
Konferensbidrag (refereegranskat)abstract
- This paper presents an interpretable machinelearning model for anomaly detection in door locks using torque data. The model aims to replace the human tactile sense in the quality control process, reducing repetitive tasks and improving reliability. The model achieved an accuracy of 96%, however, to gain social acceptance and operators' trust, interpretability of the model is crucial. The purpose of this study was to evaluate anapproach that can improve interpretability of anomalousclassifications obtained from an anomaly detection model. Weevaluate four instance-based counterfactual explanators, three of which, employ optimization techniques and one uses, a less complex, weighted nearest neighbor approach, which serve as ourbaseline. The former approaches, leverage a latent representation of the data, using a weighted principal component analysis, improving plausibility of the counter factual explanations andreduces computational cost. The explanations are presentedtogether with the 5-50-95th percentile range of the training data, acting as a frame of reference to improve interpretability. All approaches successfully presented valid and plausible counterfactual explanations. However, instance-based approachesemploying optimization techniques yielded explanations withgreater similarity to the observations and was therefore concluded to be preferable despite the higher execution times (4-16s) compared to the baseline approach (0.1s). The findings of this study hold significant value for the lock industry and can potentially be extended to other industrial settings using timeseries data, serving as a valuable point of departure for further research.
|
|
