| 1. |
- Al Sabbagh, Khaled, 1987, et al.
(author)
-
A classification of code changes and test types dependencies for improving machine learning based test selection
- 2021
-
In: SIGPLAN Notices (ACM Special Interest Group on Programming Languages). - New York, NY, USA : ACM. - 0730-8566. ; , s. 40-49
-
Conference paper (peer-reviewed)abstract
- Machine learning has been increasingly used to solve various software engineering tasks. One example of their usage is in regression testing, where a classifier is built using historical code commits to predict which test cases require execution. In this paper, we address the problem of how to link specific code commits to test types to improve the predictive performance of learning models in improving regression testing. We design a dependency taxonomy of the content of committed code and the type of a test case. The taxonomy focuses on two types of code commits: changing memory management and algorithm complexity. We reviewed the literature, surveyed experienced testers from three Swedish-based software companies, and conducted a workshop to develop the taxonomy. The derived taxonomy shows that memory management code should be tested with tests related to performance, load, soak, stress, volume, and capacity; the complexity changes should be tested with the same dedicated tests and maintainability tests. We conclude that this taxonomy can improve the effectiveness of building learning models for regression testing.
|
|
| 2. |
- Al Sabbagh, Khaled, 1987, et al.
(author)
-
Improving Data Quality for Regression Test Selection by Reducing Annotation Noise
- 2020
-
In: Proceedings - 46th Euromicro Conference on Software Engineering and Advanced Applications, SEAA 2020. ; , s. 191-194
-
Conference paper (peer-reviewed)abstract
- Big data and machine learning models have been increasingly used to support software engineering processes and practices. One example is the use of machine learning models to improve test case selection in continuous integration. However, one of the challenges in building such models is the identification and reduction of noise that often comes in large data. In this paper, we present a noise reduction approach that deals with the problem of contradictory training entries. We empirically evaluate the effectiveness of the approach in the context of selective regression testing. For this purpose, we use a curated training set as input to a tree-based machine learning ensemble and compare the classification precision, recall, and f-score against a non-curated set. Our study shows that using the noise reduction approach on the training instances gives better results in prediction with an improvement of 37% on precision, 70% on recall, and 59% on f-score.
|
|
| 3. |
- Al Sabbagh, Khaled, 1987, et al.
(author)
-
Improving Software Regression Testing Using a Machine Learning-Based Method for Test Type Selection
- 2022
-
In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). - Cham : Springer International Publishing. - 1611-3349 .- 0302-9743. ; 13709 LNCS, s. 480-496
-
Conference paper (peer-reviewed)abstract
- Since only a limited time is available for performing software regression testing, a subset of crucial test cases from the test suites has to be selected for execution. In this paper, we introduce a method that uses the relation between types of code changes and regression tests to select test types that require execution. We work closely with a large power supply company to develop and evaluate the method and measure the total regression testing time taken by our method and its effectiveness in selecting the most relevant test types. The results show that the method reduces the total regression time by an average of 18,33% when compared with the approach used by our industrial partner. The results also show that using a medium window size in the method configuration results in an improved recall rate from 61,11% to 83,33%, but not in considerable time reduction of testing. We conclude that our method can potentially be used to steer the testing effort at software development companies by guiding testers into which regression test types are essential for execution.
|
|
| 4. |
- Al-Sabbagh, Khaled, et al.
(author)
-
Improving test case selection by handling class and attribute noise
- 2022
-
In: Journal of Systems and Software. - : Elsevier BV. - 0164-1212. ; 183
-
Journal article (peer-reviewed)abstract
- Big data and machine learning models have been increasingly used to support software engineering processes and practices. One example is the use of machine learning models to improve test case selection in continuous integration. However, one of the challenges in building such models is the large volume of noise that comes in data, which impedes their predictive performance. In this paper, we address this issue by studying the effect of two types of noise, called class and attribute, on the predictive performance of a test selection model. For this purpose, we analyze the effect of class noise by using an approach that relies on domain knowledge for relabeling contradictory entries and removing duplicate ones. Thereafter, an existing approach from the literature is used to experimentally study the effect of attribute noise removal on learning. The analysis results show that the best learning is achieved when training a model on class-noise cleaned data only - irrespective of attribute noise. Specifically, the learning performance of the model reported 81% precision, 87% recall, and 84% f-score compared with 44% precision, 17% recall, and 25% f-score for a model built on uncleaned data. Finally, no causality relationship between attribute noise removal and the learning of a model for test case selection was drawn. (C) 2021 The Author(s). Published by Elsevier Inc.
|
|
| 5. |
- Al-Sabbagh, Khaled
(author)
-
Improving the Performance of Machine Learning-based Methods for Continuous Integration by Handling Noise
- 2023
-
Doctoral thesis (other academic/artistic)abstract
- Background: Modern software development companies are increasingly implementing continuous integration (CI) practices to meet market demands for delivering high-quality features. The availability of data from CI systems presents an opportunity for these companies to leverage machine learning to create methods for optimizing the CI process. Problem: The predictive performance of these methods can be hindered by inaccurate and irrelevant information – noise. Objective: The goal of this thesis is to improve the effectiveness of machine learning-based methods for CI by handling noise in data extracted from source code. Methods: This thesis employs design science research and controlled experiments to study the impact of noise-handling techniques in the context of CI. It involves developing ML-based methods for optimizing regression testing (MeBoTS and HiTTs), creating a taxonomy to reduce class noise, and implementing a class noise-handling technique (DB). Controlled experiments are carried out to examine the impact of class noise-handling on MeBoTS’ performance for CI. Results: The thesis findings show that handling class noise using the DB technique improves the performance of MeBoTS in test case selection and code change request predictions. The F1-score increases from 25% to 84% in test case selection and the Recall improved from 15% to 25% in code change request prediction after applying DB. However, handling attribute noise through a removal-based technique does not impact MeBoTS’ performance, as the F1-score remains at 66%. For memory management and complexity code changes should be tested with performance, load, soak, stress, volume, and capacity tests. Additionally, using the “majority filter” algorithm improves MCC from 0.13 to 0.58 in build outcome prediction and from -0.03 to 0.57 in code change request prediction. Conclusions: In conclusion, this thesis highlights the effectiveness of applying different class noise handling techniques to improve test case selection, build outcomes, and code change request predictions. Utilizing small code commits for training MeBoTS proves beneficial in filtering out test cases that do not reveal faults. Additionally, the taxonomy of dependencies offers an efficient and effective way for performing regression testing. Notably, handling attribute noise does not improve the predictions of test execution outcomes.
|
|
| 6. |
- Al-Sabbagh, Khaled, 1987, et al.
(author)
-
Mobile language learning applications for Arabic speaking migrants - A usability perspective
- 2018
-
In: Language Learning in Higher Education. - : Walter de Gruyter GmbH. - 2191-611X .- 2191-6128. ; 9:1, s. 71-95
-
Journal article (peer-reviewed)abstract
- Usability testing with mobile applications (apps) plays an important role in determining the ease of use of an app as well as in informing their design and development. The purpose of this research is to investigate the usability of three language learning apps with a sample of Arabic speaking migrants who have recently arrived to Sweden. In addition, We surveyed available apps for the target group on both Android and Apple market stores and used a framework for categorizing language learning apps to guide the design and development of our own app. The outcomes show that in order for users to engage in an app, there are a number of obstacles that need to be overcome to make an app motivating enough to use, such as lack of variation in functionality, instructions and feedback. Our app, Minclusion, was developed from the input obtained from the usability studies.
|
|
| 7. |
- Al-Sabbagh, Khaled
(author)
-
Noise Handling For Improving Machine Learning-Based Test Case Selection
- 2021
-
Licentiate thesis (other academic/artistic)abstract
- Background: Continuous integration is a modern software engineering practice that promotes rapid integration and testing of code changes as soon as they get committed to the project repository. One challenge in adopting this practice lies in the long time required for executing all available test cases to perform regression testing. The availability of large amounts of data about code changes and executed test cases in continuous integration systems poses an opportunity to design data-driven approaches that can effectively select a subset of test cases for regression testing. Objective: The objective of this thesis is to create a method for selecting test cases that have the highest probability of revealing faults in the system, given new code changes pushed into the code-base. Using historically committed source code and their respective executed test cases, we can utilize textual analysis and machine learning to design a method, called MeBoTs, that can learn the selection of test cases. Method: To address this objective, we carried out two design science research cycles and two controlled experiments. A combination of quantitative and qualitative data collection methods were used, including testing and code commits data, surveys, and a workshop, to evaluate and improve the effectiveness of MeBoTs in selecting effective test cases. Results: The main findings of this thesis are that: 1) using an elimination and a relabelling strategy for handling class noise in the data increases the performance of MeBoTs from 25% to 84% (F1-score), 2) eliminating attribute noise from the training data does not improve the predictive performance of a test selection model (F1-score remains unchanged at 66%), and 3) memory management changes in the source code should be tested with performance, load, soak, stress, volume, and capacity tests; the algorithmic complexity changes should be tested with the same tests for memory management code changes in addition to maintainability tests. Conclusion: Our first conclusion is that textual analysis of source code can be effective in test case selection if a class noise handling strategy is applied for curating incorrectly labeled data points in the training data. Secondly, test orchestrators do not need to handle attribute noise in the data, since it does not lead to an improvement in the performance of MeBoTs. Finally, we conclude that the performance of MeBoTs can be improved by instrumenting a tool that automatically associates code changes of specific types to test cases that are in dependency for training.
|
|
| 8. |
- Al Sabbagh, Khaled, 1987, et al.
(author)
-
Predicting build outcomes in continuous integration using textual analysis of source code commits
- 2022
-
In: PROMISE 2022 - Proceedings of the 18th International Conference on Predictive Models and Data Analytics in Software Engineering, co-located with ESEC/FSE 2022. - New York, NY, USA : ACM. ; , s. 42-51
-
Conference paper (peer-reviewed)abstract
- Machine learning has been increasingly used to solve various software engineering tasks. One example of its usage is to predict the outcome of builds in continuous integration, where a classifier is built to predict whether new code commits will successfully compile. The aim of this study is to investigate the effectiveness of fifteen software metrics in building a classifier for build outcome prediction. Particularly, we implemented an experiment wherein we compared the effectiveness of a line-level metric and fourteen other traditional software metrics on 49,040 build records that belong to 117 Java projects. We achieved an average precision of 91% and recall of 80% when using the line-level metric for training, compared to 90% precision and 76% recall for the next best traditional software metric. In contrast, using file-level metrics was found to yield a higher predictive quality (average MCC for the best software metric= 68%) than the line-level metric (average MCC= 16%) for the failed builds. We conclude that file-level metrics are better predictors of build outcomes for the failed builds, whereas the line-level metric is a slightly better predictor of passed builds.
|
|
| 9. |
- Al Sabbagh, Khaled, 1987, et al.
(author)
-
Predicting Test Case Verdicts Using TextualAnalysis of Commited Code Churns
- 2019
-
In: CEUR Workshop Proceedings. - 1613-0073. ; 2476, s. 138-153
-
Conference paper (peer-reviewed)abstract
- Background: Continuous Integration (CI) is an agile software development practice that involves producing several clean builds of the software per day. The creation of these builds involve running excessive executions of automated tests, which is hampered by high hardware cost and reduced development velocity. Goal: The goal of our research is to develop a method that reduces the number of executed test cases at each CI cycle.Method: We adopt a design research approach with an infrastructure provider company to develop a method that exploits Ma-chine Learning (ML) to predict test case verdicts for committed sourcecode. We train five different ML models on two data sets and evaluate their performance using two simple retrieval measures: precision and recall. Results: While the results from training the ML models on the first data-set of test executions revealed low performance, the curated data-set for training showed an improvement on performance with respect to precision and recall. Conclusion: Our results indicate that the method is applicable when training the ML model on churns of small sizes
|
|
| 10. |
- Al Sabbagh, Khaled, 1987, et al.
(author)
-
Selective Regression Testing based on Big Data: Comparing Feature Extraction Techniques
- 2020
-
In: IEEE Software. - 1937-4194 .- 0740-7459. ; , s. 322-329
-
Conference paper (peer-reviewed)abstract
- Regression testing is a necessary activity in continuous integration (CI) since it provides confidence that modified parts of the system are correct at each integration cycle. CI provides large volumes of data which can be used to support regression testing activities. By using machine learning, patterns about faulty changes in the modified program can be induced, allowing test orchestrators to make inferences about test cases that need to be executed at each CI cycle. However, one challenge in using learning models lies in finding a suitable way for characterizing source code changes and preserving important information. In this paper, we empirically evaluate the effect of three feature extraction algorithms on the performance of an existing ML-based selective regression testing technique. We designed and performed an experiment to empirically investigate the effect of Bag of Words (BoW), Word Embeddings (WE), and content-based feature extraction (CBF). We used stratified cross validation on the space of features generated by the three FE techniques and evaluated the performance of three machine learning models using the precision and recall metrics. The results from this experiment showed a significant difference between the models' precision and recall scores, suggesting that the BoW-fed model outperforms the other two models with respect to precision, whereas a CBF-fed model outperforms the rest with respect to recall.
|
|
