| 1. |
- Heckmann Barbalho de Figueroa, Laiz, et al.
(författare)
-
A Modeling Approach for Bioinformatics Workflows
- 2019
-
Ingår i: The Practice of Enterprise Modeling - 12th {IFIP} Working Conference, PoEM 2019, Luxembourg, Luxembourg, November 27-29, 2019, Proceedings. - Cham : Springer.
-
Konferensbidrag (refereegranskat)
|
|
| 2. |
- Ahmad, Azeem, et al.
(författare)
-
A Multi-factor Approach for Flaky Test Detection and Automated Root Cause Analysis
- 2021
-
Ingår i: Proceedings - Asia-Pacific Software Engineering Conference, APSEC. - : IEEE COMPUTER SOC. - 1530-1362. ; , s. 338-348
-
Konferensbidrag (refereegranskat)abstract
- Developers often spend time to determine whether test case failures are real failures or flaky. The flaky tests, also known as non-deterministic tests, switch their outcomes without any modification in the codebase, hence reducing the confidence of developers during maintenance as well as in the quality of a product. Re-running test cases to reveal flakiness is resource-consuming, unreliable and does not reveal the root causes of test flakiness. Our paper evaluates a multi-factor approach to identify flaky test executions implemented in a tool named MDF laker. The four factors are: trace-back coverage, flaky frequency, number of test smells, and test size. Based on the extracted factors, MDFlaker uses k-Nearest Neighbor (KNN) to determine whether failed test executions are flaky. We investigate MDFlaker in a case study with 2166 test executions from different open-source repositories. We evaluate the effectiveness of our flaky detection tool. We illustrate how the multi-factor approach can be used to reveal root causes for flakiness, and we conduct a qualitative comparison between MDF laker and other tools proposed in literature. Our results show that the combination of different factors can be used to identify flaky tests. Each factor has its own trade-off, e.g., trace-back leads to many true positives, while flaky frequency yields more true negatives. Therefore, specific combinations of factors enable classification for testers with limited information (e.g., not enough test history information).
|
|
| 3. |
- Ahmad, Azeem, et al.
(författare)
-
An Industrial Study on the Challenges and Effects of Diversity-Based Testing in Continuous Integration
- 2023
-
Ingår i: IEEE International Conference on Software Quality, Reliability and Security, QRS. - 2693-9177. - 9798350319583
-
Konferensbidrag (refereegranskat)abstract
- Many test prioritisation techniques have been proposed in order to improve test effectiveness of Continuous Integration (CI) pipelines. Particularly, diversity-based testing (DBT) has shown promising and competitive results to improve test effectiveness. However, the technical and practical challenges of introducing test prioritisation in CI pipelines are rarely discussed, thus hindering the applicability and adoption of those proposed techniques. This research builds on our prior work in which we evaluated diversity-based techniques in an industrial setting. This work investigates the factors that influence the adoption of DBT both in connection to improvements in test cost-effectiveness, as well as the process and human related challenges to transfer and use DBT prioritisation in CI pipelines. We report on a case study considering the CI pipeline of Axis Communications in Sweden. We performed a thematic analysis of a focus group interview with senior practitioners at the company to identify the challenges and perceived benefits of using test prioritisation in their test process. Our thematic analysis reveals a list of ten challenges and seven perceived effects of introducing test prioritisation in CI cycles. For instance, our participants emphasized the importance of introducing comprehensible and transparent techniques that instill trust in its users. Moreover, practitioners prefer techniques compatible with their current test infrastructure (e.g., test framework and environments) in order to reduce instrumentation efforts and avoid disrupting their current setup. In conclusion, we have identified tradeoffs between different test prioritisation techniques pertaining to the technical, process and human aspects of regression testing in CI. We summarize those findings in a list of seven advantages that refer to specific stakeholder interests and describe the effects of adopting DBT in CI pipelines.
|
|
| 4. |
- Cvitic, Petya Hristova, et al.
(författare)
-
Investigating Software Testing and Maintenance of Open-Source Distributed Ledger
- 2023
-
Ingår i: Proceedings - 2023 IEEE International Conference on Software Analysis, Evolution and Reengineering, SANER 2023. - : IEEE conference proceedings. - 9781665452786 ; , s. 886-896
-
Konferensbidrag (refereegranskat)abstract
- A distributed ledger is the backbone of all blockchain solutions. It provides a shared database spreading across a network of nodes. The number of DL solutions and their implementations has grown in recent years. Besides the architectural and performance promises of these solutions, organizations seekingto implement DL also need to consider the overall quality of the software available and its ecosystem. Particularly, previous research has identified the need to better understand the testing and maintenance practices behind these types of technologies. This paper investigates the testing and maintenance of 18 different open-source projects that implement distributed ledgers. We perform a manual inspection of test artefacts and mine the history of commits, issues and contributors of the chosen projects to understand the landscape of testing and maintenance in these projects. Our findings suggest that unit and integration tests are present in most projects, they do not follow a holistic system testing approach. Moreover, projects rely on a small team of core contributors (5 on average). While the projects are continuously maintained, larger changes are uncommon. Our results can be used for benchmarking and pinpointing areas of improvement for the development of distributed ledgers.
|
|
| 5. |
- Cvitic, Petya Hristova, et al.
(författare)
-
Investigating Software Testing and Maintenance of Open-Source Distributed Ledger
- 2023
-
Ingår i: Proceedings - 2023 IEEE International Conference on Software Analysis, Evolution and Reengineering, SANER 2023. - : IEEE. - 2833-9061. - 9781665452786
-
Konferensbidrag (refereegranskat)abstract
- A distributed ledger is the backbone of all blockchain solutions. It provides a shared database spreading across a network of nodes. The number of DL solutions and their implementations has grown in recent years. Besides the architectural and performance promises of thesesolutions, organizations seekingto implement DL also need to consider the overall quality of the software available and its ecosystem. Particularly, previous research has identified the need to better understand the testing and maintenance practices behind these types of technologies. This paper investigates the testing and maintenance of 18 different open-source projects that implement distributed ledgers. We perform a manual inspection of test artefacts and mine the history of commits, issues and contributors of the chosen projects to understand the landscape of testing and maintenance in these projects. Our findings suggest that unit and integration tests are present in most projects, they do not follow a holistic system testing approach. Moreover, projects rely on a small team of core contributors (5 on average). While the projects are continuously maintained, larger changes are uncommon. Our results can be used for benchmarking and pinpointing areas of improvement for the development of distributed ledgers.
|
|
| 6. |
- de Oliveira Neto, Francisco Gomes, et al.
(författare)
-
Building Collaborative Learning : Exploring Social Annotation in Introductory Programming
- 2024
-
Ingår i: Proceedings - International Conference on Software Engineering. - : IEEE Computer Society. - 9798400704987 ; , s. 12-21
-
Konferensbidrag (refereegranskat)abstract
- The increasing demand for software engineering education presents learning challenges in courses due to the diverse range of topics that require practical applications, such as programming or software design, all of which are supported by group work and interaction. Social Annotation (SA) is an approach to teaching that can enhance collaborative learning among students. In SA, both students and teachers utilize platforms like Feedback Fruits, Perusall, and Diigo to collaboratively annotate and discuss course materials. This approach encourages students to share their thoughts and answers with their peers, fostering a more interactive learning environment. We share our experience of implementing social annotation via Perusall as a preparatory tool for lectures in an introductory programming course aimed at undergraduate students in Software Engineering. We report the impact of Perusall on the examination results of 112 students. Our results show that 81% of students engaged in meaningful social annotation successfully passed the course. Notably, the proportion of students passing the exam tends to rise as they complete more Perusall assignments. In contrast, only 56% of students who did not participate in Perusall discussions managed to pass the exam. We did not enforce mandatory Perusall participation in the course. Yet, the feedback from our course evaluation questionnaire reveals that most students ranked Perusall among their favorite components of the course and that their interest in the subject has increased.
|
|
| 7. |
- de Oliveira Neto, Francisco Gomes, et al.
(författare)
-
Challenges of Aligning Requirements Engineering and System Testing in Large-Scale Agile: A Multiple Case Study
- 2017
-
Ingår i: 2017 IEEE 25th International Requirements Engineering Conference Workshops (REW), 4-8 Sept. 2017, Lisbon, Portugal. - : IEEE. - 9781538634882
-
Konferensbidrag (refereegranskat)abstract
- Abstract: As agile methods become more pervasive, agile practices are applied to more large-scale systems with a scope that goes beyond pure software. The expansion of agile in these contexts provides benefits, but creates new challenges. Widespread use of agile has changed the way we must think about practices both in Requirements Engineering (RE) and in System Testing (ST). Our experience shows that many challenges in the application of large-scale agile development relate to either RE or ST, and in particular to the alignment between these areas. In this paper we present large-scale agile-related challenges from a multiple case study which relate to REST alignment. We map our challenges to an existing framework for REST alignment, and make an initial attempt to suggest agile RE practices from the literature which may alleviate these challenges. Our results show that the interviewed companies need to first adopt more agile RE practices to enhance REST alignment and then leverage agile testing. Future work will look more towards evaluating these best practices.
|
|
| 8. |
- de Oliveira Neto, Francisco Gomes, et al.
(författare)
-
Full modification coverage through automatic similarity-based test case selection
- 2016
-
Ingår i: Information and Software Technology. - : Elsevier BV. - 0950-5849. ; 80, s. 124-137
-
Tidskriftsartikel (refereegranskat)abstract
- Context: This paper presents the similarity approach for regression testing (SART), where a similarity-based test case selection technique is used in a model-based testing process to provide selection of test cases exercising modified parts of a specification model. Unlike other model-based regression testing techniques, SART relies on similarity analysis among test cases to identify modifications, instead of comparing models, hence reducing the dependency on specific types of model. Objective: To present convincing evidence of the usage of similarity measures for modification-traversing test case selection. Method: We investigate SART in a case study and an experiment. The case study uses artifacts from industry and should be seen as a sanity check of SART, while the experiment focuses on gaining statistical power through the generation of synthetical models in order to provide convincing evidence of SART’s effectiveness. Through posthoc analysis we obtain p-values and effect sizes to observe statistically significant differences between treatments with respect to transition and modification coverage. Results: The case study with industrial artifacts revealed that SART is able to uncover the same number of defects as known similarity-based test case selection techniques. In turn, the experiment shows that SART, unlike the other investigated techniques, presents 100% modification coverage. In addition, all techniques covered a similar percentage of model transitions. Conclusions: In summary, not only does SART provide transition and defect coverage equal to known STCS techniques, but it exceeds greatly in covering modified parts of the specification model, being a suitable candidate for model-based regression testing. Keywords: Regression testing, Test case selection, Model-based testing, Experimental Study
|
|
| 9. |
- de Oliveira Neto, Francisco Gomes, et al.
(författare)
-
Improving continuous integration with similarity-based test case selection
- 2018
-
Ingår i: Proceedings of the 13th International Workshop on Automation of Software Test. - New York : ACM Digital Library. - 0270-5257. - 9781450357432 ; , s. 39-45
-
Konferensbidrag (refereegranskat)abstract
- Automated testing is an essential component of Continuous Integration (CI) and Delivery (CD), such as scheduling automated test sessions on overnight builds. That allows stakeholders to execute entire test suites and achieve exhaustive test coverage, since running all tests is often infeasible during work hours, i.e., in parallel to development activities. On the other hand, developers also need test feedback from CI servers when pushing changes, even if not all test cases are executed. In this paper we evaluate similarity-based test case selection (SBTCS) on integration-level tests executed on continuous integration pipelines of two companies. We select test cases that maximise diversity of test coverage and reduce feedback time to developers. Our results confirm existing evidence that SBTCS is a strong candidate for test optimisation, by reducing feedback time (up to 92% faster in our case studies) while achieving full test coverage using only information from test artefacts themselves.
|
|
| 10. |
- de Oliveira Neto, Francisco Gomes, et al.
(författare)
-
Searching for models to evaluate software technology
- 2013
-
Ingår i: 2013 1st International Workshop on Combining Modelling and Search-Based Software Engineering, CMSBSE 2013 - Proceedings. - 9781467362849 ; , s. 12-15
-
Konferensbidrag (refereegranskat)abstract
- Modeling and abstraction is key in all engineering processes and have found extensive use also in software engineering. When developing new methodologies and techniques to support software engineers we want to evaluate them on realistic models. However, this is a challenge since (1) it is hard to get industry to give access to their models, and (2) we need a large number of models to systematically evaluate a technology. This paper proposes that search-based techniques can be used to search for models with desirable properties, which can then be used to systematically evaluate model-based technologies. By targeting properties seen in industrial models we can then get the best of both worlds: models that are similar to models used in industry but in quantities that allow extensive experimentation. To exemplify our ideas we consider a specific case in which a model generator is used to create models to test a regression test optimization technique. © 2013 IEEE.
|
|
| 11. |
- de Oliveira Neto, Francisco Gomes, et al.
(författare)
-
Visualisation to Support Fault Localisation in Distributed Embedded Systems within the Automotive Industry
- 2018
-
Ingår i: 2018 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW). - : IEEE. - 9781538694435
-
Konferensbidrag (refereegranskat)abstract
- This paper presents the design, development and evaluation of a software tool to assist the localisation of root causes of test case failures in distributed embedded systems, specifically vehicle systems controlled by a network of electronic control units (ECUs). We use data visualising to provide sensible information from a large number of test execution logs from large-scale software integration testing under a continuous integration process. Our goal is to allow more efficient root-cause identification of failures and foster a continuous feedback loop in the fault localisation process. We evaluate our solution in-situ at the Research and Development division of Volvo Car Corporation (VCC). Our prototype helps the failure debugging procedures by presenting clear and concise data and by allowing stakeholders to filter and control which information is displayed. Moreover, it encourages a systematic and continuous analysis of the current state of testing by aggregating and categorising historical data from test harnesses to identify patterns and trends in test results.
|
|
| 12. |
- Dobslaw, Felix, 1983-, et al.
(författare)
-
Automated black-box boundary value detection
- 2023
-
Ingår i: PeerJ Computer Science. - : PeerJ. - 2376-5992. ; 9
-
Tidskriftsartikel (refereegranskat)abstract
- Abstract Software systems typically have an input domain that can be subdivided into sub-domains, each of which generates similar or related outputs. Testing it on the boundaries between these sub-domains is critical to ensure high-quality software. Therefore, boundary value analysis and testing have been a fundamental part of the software testing toolbox for a long time and are typically taught early to software engineering students. Despite its many argued benefits, boundary value analysis for a given software specification or application is typically described in abstract terms. This allows for variation in how testers apply it and in the benefits they see. Additionally, its adoption has been limited since it requires a specification or model to be analysed. We propose an automated black-box boundary value detection method to support software testers in performing systematic boundary value analysis. This dynamic method can be utilized even without a specification or model. The proposed method is based on a metric referred to as the program derivative, which quantifies the level of boundariness of test inputs. By combining this metric with search algorithms, we can identify and rank pairs of inputs as good boundary candidates, i.e., inputs that are in close proximity to each other but with outputs that are far apart. We have implemented the AutoBVA approach and evaluated it on a curated dataset of example programs. Furthermore, we have applied the approach broadly to a sample of 613 functions from the base library of the Julia programming language. The approach could identify boundary candidates that highlight diverse boundary behaviours in over 70% of investigated systems under test. The results demonstrate that even a simple variant of the program derivative, combined with broad sampling and search over the input space, can identify interesting boundary candidates for a significant portion of the functions under investigation. In conclusion, we also discuss the future extension of the approach to encompass more complex systems under test cases and datatypes.
|
|
| 13. |
- Dobslaw, Felix, et al.
(författare)
-
Automated black-box boundary value detection
- 2023
-
Ingår i: PeerJ Computer Science. - 2376-5992. ; 9
-
Tidskriftsartikel (refereegranskat)abstract
- Software systems typically have an input domain that can be subdivided into subdomains, each of which generates similar or related outputs. Testing it on the boundaries between these sub-domains is critical to ensure high-quality software. Therefore, boundary value analysis and testing have been a fundamental part of the software testing toolbox for a long time and are typically taught early to software engineering students. Despite its many argued benefits, boundary value analysis for a given software specification or application is typically described in abstract terms. This allows for variation in how testers apply it and in the benefits they see. Additionally, its adoption has been limited since it requires a specification or model to be analysed. We propose an automated black-box boundary value detection method to support software testers in performing systematic boundary value analysis. This dynamic method can be utilized even without a specification or model. The proposed method is based on a metric referred to as the program derivative, which quantifies the level of boundariness of test inputs. By combining this metric with search algorithms, we can identify and rank pairs of inputs as good boundary candidates, i.e., inputs that are in close proximity to each other but with outputs that are far apart. We have implemented the AutoBVA approach and evaluated it on a curated dataset of example programs. Furthermore, we have applied the approach broadly to a sample of 613 functions from the base library of the Julia programming language. The approach could identify boundary candidates that highlight diverse boundary behaviours in over 70% of investigated systems under test. The results demonstrate that even a simple variant of the program derivative, combined with broad sampling and search over the input space, can identify interesting boundary candidates for a significant portion of the functions under investigation. In conclusion, we also discuss the future extension of the approach to encompass more complex systems under test cases and datatypes.
|
|
| 14. |
- Dobslaw, Felix, et al.
(författare)
-
Automated black-box boundary value detection
- 2023
-
Ingår i: PeerJ Computer Science. - 2376-5992. ; 9
-
Tidskriftsartikel (refereegranskat)abstract
- Software systems typically have an input domain that can be subdivided into subdomains, each of which generates similar or related outputs. Testing it on the boundaries between these sub-domains is critical to ensure high-quality software. Therefore, boundary value analysis and testing have been a fundamental part of the software testing toolbox for a long time and are typically taught early to software engineering students. Despite its many argued benefits, boundary value analysis for a given software specification or application is typically described in abstract terms. This allows for variation in how testers apply it and in the benefits they see. Additionally, its adoption has been limited since it requires a specification or model to be analysed. We propose an automated black-box boundary value detection method to support software testers in performing systematic boundary value analysis. This dynamic method can be utilized even without a specification or model. The proposed method is based on a metric referred to as the program derivative, which quantifies the level of boundariness of test inputs. By combining this metric with search algorithms, we can identify and rank pairs of inputs as good boundary candidates, i.e., inputs that are in close proximity to each other but with outputs that are far apart. We have implemented the AutoBVA approach and evaluated it on a curated dataset of example programs. Furthermore, we have applied the approach broadly to a sample of 613 functions from the base library of the Julia programming language. The approach could identify boundary candidates that highlight diverse boundary behaviours in over 70% of investigated systems under test. The results demonstrate that even a simple variant of the program derivative, combined with broad sampling and search over the input space, can identify interesting boundary candidates for a significant portion of the functions under investigation. In conclusion, we also discuss the future extension of the approach to encompass more complex systems under test cases and datatypes.
|
|
| 15. |
- Dobslaw, F., et al.
(författare)
-
Estimating Return on Investment for GUI Test Automation Frameworks
- 2019
-
Ingår i: 2019 IEEE 30th International Symposium on Software Reliability Engineering (ISSRE). - : IEEE. - 9781728149820
-
Konferensbidrag (refereegranskat)abstract
- Automated graphical user interface (GUI) tests can reduce manual testing activities and increase test frequency. This motivates the conversion of manual test cases into automated GUI tests. However, it is not clear whether such automation is cost-effective given that GUI automation scripts add to the code base and demand maintenance as a system evolves. In this paper, we introduce a method for estimating maintenance cost and Return on Investment (ROI) for Automated GUI Testing (AGT). The method utilizes the existing source code change history and has the potential to be used for the evaluation of other testing or quality assurance automation technologies. We evaluate the method for a real-world, industrial software system and compare two fundamentally different AGT frameworks, namely Selenium and EyeAutomate, to estimate and compare their ROI. We also report on their defect-finding capabilities and usability. The quantitative data is complemented by interviews with employees at the company the study has been conducted at. The method was successfully applied, and estimated maintenance cost and ROI for both frameworks are reported. Overall, the study supports earlier results showing that implementation time is the leading cost for introducing AGT. The findings further suggest that, while EyeAutomate tests are significantly faster to implement, Selenium tests require more of a programming background but less maintenance.
|
|
| 16. |
|
|
| 17. |
- Erlenhov, Linda, 1979, et al.
(författare)
-
An empirical study of bots in software development: Characteristics and challenges from a practitioner's perspective
- 2020
-
Ingår i: ESEC/FSE 2020 - Proceedings of the 28th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering. - New York, NY, USA : ACM.
-
Konferensbidrag (refereegranskat)abstract
- © 2020 ACM. Software engineering bots - automated tools that handle tedious tasks - are increasingly used by industrial and open source projects to improve developer productivity. Current research in this area is held back by a lack of consensus of what software engineering bots (DevBots) actually are, what characteristics distinguish them from other tools, and what benefits and challenges are associated with DevBot usage. In this paper we report on a mixed-method empirical study of DevBot usage in industrial practice. We report on findings from interviewing 21 and surveying a total of 111 developers. We identify three different personas among DevBot users (focusing on autonomy, chat interfaces, and "smartness"), each with different definitions of what a DevBot is, why developers use them, and what they struggle with.We conclude that future DevBot research should situate their work within our framework, to clearly identify what type of bot the work targets, and what advantages practitioners can expect. Further, we find that there currently is a lack of general purpose "smart"bots that go beyond simple automation tools or chat interfaces. This is problematic, as we have seen that such bots, if available, can have a transformative effect on the projects that use them.
|
|
| 18. |
|
|
| 19. |
- Felderer, Michael, 1978-, et al.
(författare)
-
A testability analysis framework for non-functional properties
- 2018
-
Ingår i: 2018 IEEE 11TH INTERNATIONAL CONFERENCE ON SOFTWARE TESTING, VERIFICATION AND VALIDATION WORKSHOPS (ICSTW). - : Institute of Electrical and Electronics Engineers Inc.. - 9781538663523 ; , s. 54-58
-
Konferensbidrag (refereegranskat)abstract
- This paper presents background, the basic steps and an example for a testability analysis framework for non-functional properties.
|
|
| 20. |
- Fontes, Afonso, 1987, et al.
(författare)
-
Automated Support forUnit Test Generation
- 2023
-
Ingår i: Natural Computing Series. - Singapore : Springer. - 1619-7127. ; , s. 179-219
-
Bokkapitel (refereegranskat)abstract
- Unit testing is a stage of testing where the smallest segment of code that can be tested in isolation from the rest of the system—often a class—is tested. Unit tests are typically written as executable code, often in a format provided by a unit testing framework such as pytest for Python. Creating unit tests is a time and effort-intensive process with many repetitive, manual elements. To illustrate how AI can support unit testing, this chapter introduces the concept of search-based unit test generation. This technique frames the selection of test input as an optimization problem—we seek a set of test cases that meet some measurable goal of a tester—and unleashes powerful metaheuristic search algorithms to identify the best possible test cases within a restricted timeframe. This chapter introduces two algorithms that can generate pytest-formatted unit tests, tuned towards coverage of source code statements. The chapter concludes by discussing more advanced concepts and gives pointers to further reading for how artificial intelligence can support developers and testers when unit testing software.
|
|
| 21. |
- Kasauli, Rashida, 1984, et al.
(författare)
-
Requirements engineering challenges and practices in large-scale agile system development
- 2022
-
Ingår i: Accelerating Digital Transformation: 10 Years of Software Center. - 9783031108730 ; , s. 293-350
-
Bokkapitel (övrigt vetenskapligt/konstnärligt)abstract
- Context: Agile methods have become mainstream even in large-scale systems engineering companies that need to accommodate different development cycles of hardware and software. For such companies, requirements engineering is an essential activity that involves upfront and detailed analysis which can be at odds with agile development methods. Objective: This paper presents a multiple case study with seven large-scale systems companies, reporting their challenges, together with best practices from industry. We also analyse literature about two popular large-scale agile frameworks, SAFe® and LeSS, to derive potential solutions for the challenges. Method: Our results are based on 20 qualitative interviews, five focus groups, and eight cross-company workshops which we used to both collect and validate our results. Results: We found 24 challenges which we grouped in six themes, then mapped to solutions from SAFe®, LeSS, and our companies, when available. Conclusion: In this way, we contribute a comprehensive overview of RE challenges in relation to large-scale agile system development, evaluate the degree to which they have been addressed, and outline research gaps. We expect these results to be useful for practitioners who are responsible for designing processes, methods, or tools for large scale agile development as well as guidance for researchers.
|
|
| 22. |
- Michael Ayas, Hamdy, 1994, et al.
(författare)
-
An Empirical Analysis of Microservices Systems Using Consumer-Driven Contract Testing
- 2022
-
Ingår i: Proceedings - 48th Euromicro Conference on Software Engineering and Advanced Applications, SEAA 2022. ; , s. 92-99
-
Konferensbidrag (refereegranskat)abstract
- Testing has a prominent role in revealing faults in software based on microservices. One of the most important discussion points in MSAs is the granularity of services, often in different levels of abstraction. Similarly, the granularity of tests in MSAs is reflected in different test types. However, it is challenging to conceptualize how the overall testing architecture comes together when combining testing in different levels of abstraction for microservices. There is no empirical evidence on the overall testing architecture in such microservices implementations. Furthermore, there is a need to empirically understand how the current state of practice resonates with existing best practices on testing. In this study, we mine Github to find different candidate projects for an in-depth, qualitative assessment of their test artifacts. We analyze 16 repositories that use microservices and include various test artifacts. We focus on four projects that use consumer-driven-contract testing. Our results demonstrate how these projects cover different levels of testing. This study (i) drafts a testing architecture including activities and artifacts, and (ii) demonstrates how these align with best practices and guidelines. Our proposed architecture helps the categorization of system and test artifacts in empirical studies of microservices. Finally, we showcase a view of the boundaries between different levels of testing in systems using microservices.
|
|
| 23. |
- Stikkolorum, Dave, et al.
(författare)
-
Evaluating didactic approaches used by teaching assistants for software analysis and design using UML
- 2018
-
Ingår i: Proceedings of the 3rd European Conference of Software Engineering Education. - New York, NY, USA : ACM. - 9781450363839
-
Konferensbidrag (refereegranskat)abstract
- © 2018 Copyright held by the owner/author(s). Teaching assistants (TAs) are employed for supporting various activities of a course, from assisting with course material, grading and, predominantly, bridging the interaction between teachers and students. This role is even more significant in large courses with dozens of students, particularly during lab sessions with practical assignments. Therefore, here we explore how does the role of the TA support students to overcome the challenges they face when learning software design. During a bachelor course on Analysis and Design, we collected data from i) weekly interviews with TAs, ii) student's assignment hand-ins and iii) the corresponding feedback reports written by TAs to students. By analysing these three different sources we propose guidelines and practices for effective deployment of TAs.
|
|
| 24. |
- Torkar, Richard, 1971, et al.
(författare)
-
A Method to Assess and Argue for Practical Significance in Software Engineering
- 2022
-
Ingår i: IEEE Transactions on Software Engineering. - : IEEE Computer Society. - 0098-5589 .- 1939-3520. ; 48:6, s. 2053-2065
-
Tidskriftsartikel (refereegranskat)abstract
- A key goal of empirical research in software engineering is to assess practical significance, which answers the question whether the observed effects of some compared treatments show a relevant difference in practice in realistic scenarios. Even though plenty of standard techniques exist to assess statistical significance, connecting it to practical significance is not straightforward or routinely done; indeed, only a few empirical studies in software engineering assess practical significance in a principled and systematic way. In this paper, we argue that Bayesian data analysis provides suitable tools to assess practical significance rigorously. We demonstrate our claims in a case study comparing different test techniques. The case study's data was previously analyzed (Afzal et al., 2015) using standard techniques focusing on statistical significance. Here, we build a multilevel model of the same data, which we fit and validate using Bayesian techniques. Our method is to apply cumulative prospect theory on top of the statistical model to quantitatively connect our statistical analysis output to a practically meaningful context. This is then the basis both for assessing and arguing for practical significance. Our study demonstrates that Bayesian analysis provides a technically rigorous yet practical framework for empirical software engineering. A substantial side effect is that any uncertainty in the underlying data will be propagated through the statistical model, and its effects on practical significance are made clear. Thus, in combination with cumulative prospect theory, Bayesian analysis supports seamlessly assessing practical significance in an empirical software engineering context, thus potentially clarifying and extending the relevance of research for practitioners.
|
|
| 25. |
- Torkar, Richard, 1971, et al.
(författare)
-
Statistical and practical significance of empirical software engineering research: A maturity model
- 2017
-
Tidskriftsartikel (övrigt vetenskapligt/konstnärligt)abstract
- Software engineering research is maturing and papers increasingly support their arguments with empirical data from a multitude of sources, using statistical tests to judge if and to what degree empirical evidence supports their hypotheses. This paper presents trends and current state of art regarding statistical maturity in empirical software engineering research; the objective being the development of a statistical maturity model. First, we manually reviewed papers from four well-known and top ranked journals producing a review protocol along with the view of current (2015) state of art concerning statistical maturity, practical significance and reproducibility of empirical software engineering research. Our protocol was then used as ground truth, i.e., training set, for a semi-automatic classification of studies for the years 2001--2015 using a total of 3,011 papers. We used the extracted data to develop a statistical maturity model which also includes a model for how to argue for practical significance. The statistical maturity of empirical software engineering research has an upward trend in certain areas (e.g., use of nonparametric statistics, but also more generally in the usage of quantitative analysis). However, we also see how our research area currently often fails to connect the statistical analysis to practical significance. For instance, we argue that conclusions should explicitly state contributions to software engineering practice, e.g., the return on investment for practitioners. We argue that the statistical maturity model can be used by researchers and practitioners to build a coherent statistical analysis and guide them in the choice of statistical approaches of its steps. The final goal for a researcher would be to, in a clearer way, present and argue for the practical significance of their findings. Bayesian analysis, we believe, has a role to play in this.
|
|
