| 1. |
- Baldassarre, Maria Teresa, et al.
(författare)
-
Studying test-driven development and its retainment over a six-month time span
- 2021
-
Ingår i: Journal of Systems and Software. - : Elsevier Inc.. - 0164-1212 .- 1873-1228. ; 176
-
Tidskriftsartikel (refereegranskat)abstract
- In this paper, we investigate the effect of TDD, as compared to a non-TDD approach, as well as its retainment (or retention) over a time span of (about) six months. To pursue these objectives, we conducted a (quantitative) longitudinal cohort study with 30 novice developers (i.e., third-year undergraduate students in Computer Science). We observed that TDD affects neither the external quality of software products nor developers’ productivity. However, we observed that the participants applying TDD produced significantly more tests, with a higher fault-detection capability, than those using a non-TDD approach. As for the retainment of TDD, we found that TDD is retained by novice developers for at least six months. © 2021 Elsevier Inc.
|
|
| 2. |
- Falessi, Davide, et al.
(författare)
-
Empirical software engineering experts on the use of students and professionals in experiments
- 2018
-
Ingår i: Empirical Software Engineering. - : Springer New York LLC. - 1382-3256 .- 1573-7616. ; 23:1, s. 452-489
-
Tidskriftsartikel (refereegranskat)abstract
- [Context] Controlled experiments are an important empirical method to generate and validate theories. Many software engineering experiments are conducted with students. It is often claimed that the use of students as participants in experiments comes at the cost of low external validity while using professionals does not. [Objective] We believe a deeper understanding is needed on the external validity of software engineering experiments conducted with students or with professionals. We aim to gain insight about the pros and cons of using students and professionals in experiments. [Method] We performed an unconventional, focus group approach and a follow-up survey. First, during a session at ISERN 2014, 65 empirical researchers, including the seven authors, argued and discussed the use of students in experiments with an open mind. Afterwards, we revisited the topic and elicited experts’ opinions to foster discussions. Then we derived 14 statements and asked the ISERN attendees excluding the authors, to provide their level of agreement with the statements. Finally, we analyzed the researchers’ opinions and used the findings to further discuss the statements. [Results] Our survey results showed that, in general, the respondents disagreed with us about the drawbacks of professionals. We, on the contrary, strongly believe that no population (students, professionals, or others) can be deemed better than another in absolute terms. [Conclusion] Using students as participants remains a valid simplification of reality needed in laboratory contexts. It is an effective way to advance software engineering theories and technologies but, like any other aspect of study settings, should be carefully considered during the design, execution, interpretation, and reporting of an experiment. The key is to understand which developer population portion is being represented by the participants in an experiment. Thus, a proposal for describing experimental participants is put forward.
|
|
| 3. |
|
|
| 4. |
- Ghazi, Ahmad Nauman, 1983-
(författare)
-
Structuring Exploratory Testing through Test Charter Design and Decision Support
- 2017
-
Doktorsavhandling (övrigt vetenskapligt/konstnärligt)abstract
- Context: Exploratory testing (ET) is an approach to test software with a strong focus on personal skills and freedom of the tester. ET emphasises the simultaneous design and execution of tests with minimal test documentation. Test practitioners often claim that their choice to use ET as an important alternative to scripted testing is based on several benefits ET exhibits over the scripted testing. However, these claims lack empirical evidence as there is little research done in this area. Moreover, ET is usually considered an ad-hoc way of doing testing as everyone does it differently. There have been some attempts in past to provide structure to ET. Session based test management (SBTM) is an approach that attempts to provide some structure to ET and gives some basic guidelines to structuring the test sessions. However, these guidelines are still very abstract and are very open to individuals' interpretation.Objective: The main objective of this doctoral thesis is to support practitioners in their decisions about choosing exploratory versus scripted testing. Furthermore, it is also aimed to investigate the empirical evidence in support of ET and find ways to structure ET and classify different levels of exploration that drive the choices made by exploratory testers. Another objective of this thesis is to provide a decision support system to select levels of exploration in overall test process.Method: The findings presented in this thesis are obtained through a controlled experiment with participants from industry and academia, exploratory surveys, interviews and focus groups conducted at different companies including Ericsson AB, Sony Mobile Communications, Axis Communications AB and Softhouse Consulting Baltic AB.Results: Using the exploratory survey, we found three test techniques to be most relevant in context of testing software systems and in particular heterogeneous systems. The most frequently used technique mentioned by the practitioners is ET which is not a much researched topic. We also found many interesting claims about ET in grey literature produced by practitioners in the form of informal presentations and blogs but these claims lacked any empirical evidence. Therefore, a controlled experiment was conducted with students and industry practitioners to compare ET with scripted testing. The experiment results show that ET detects significantly more critical defects compared to scripted testing and is more time efficient. However, ET has its own limitations and there is not a single way to use it for testing. In order to provide structure to ET, we conducted a study where we propose checklists to support test charter design in ET. Furthermore, two more industrial focus group studies at four companies were conducted that resulted in a taxonomy of exploration levels in ET and a decision support method for selecting exploration levels in ET. Lastly, we investigated different problems that researchers face when conducting surveys in software engineering and have presented mitigation strategies for these problems.Conclusion: The taxonomy for levels of exploration in ET, proposed in this thesis, provided test practitioners at the companies a better understanding of the underlying concepts of ET and a way to structure their test charters. A number of influence factors elicited as part of this thesis also help them prioritise which level of exploration suits more to their testing in the context of their products. Furthermore, the decision support method provided the practitioners to reconsider their current test focus to test their products in a more effective way.
|
|
| 5. |
- Romano, Simone, et al.
(författare)
-
On researcher bias in Software Engineering experiments
- 2021
-
Ingår i: Journal of Systems and Software. - : Elsevier Inc.. - 0164-1212 .- 1873-1228. ; 182
-
Tidskriftsartikel (refereegranskat)abstract
- Researcher bias occurs when researchers influence the results of an empirical study based on their expectations, either consciously or unconsciously. Researcher bias might be due to the use of Questionable Research Practices (QRPs). In research fields like medicine, blinding techniques have been applied to counteract researcher bias. In this paper, we present two studies to increase our body of knowledge on researcher bias in Software Engineering (SE) experiments, including: (i) QRPs potentially leading to researcher bias; (ii) causes behind researcher bias; and (iii) possible actions to counteract researcher bias with a focus on, but not limited to, blinding techniques. The former is an interview study, intended as an exploratory study, with nine experts of the empirical SE community. The latter is a quantitative survey with 51 respondents, who were experts of the above-mentioned community. The findings from the exploratory study represented the starting point to design the survey. In particular, we defined the questionnaire of this survey to support the findings from the exploratory study. From the interview study, it emerged that some QRPs (e.g., post-hoc outlier criteria) are acceptable in certain cases. Also, it appears that researcher bias is perceived in SE and, to counteract researcher bias, a number of solutions have been highlighted. For example, duplicating the data analysis in SE experiments or fostering open data policies in SE conferences/journals. The findings from the interview study are mostly confirmed by those from the survey, and allowed us to delineate recommendations to counteract researcher bias in SE experiments. Some recommendations are intended for SE researchers, while others are purposeful for the boards of SE research venues. © 2021 Elsevier Inc.
|
|
| 6. |
- Romano, Simone, et al.
(författare)
-
Researcher Bias in Software Engineering Experiments : A Qualitative Investigation
- 2020
-
Ingår i: Proceedings - 46th Euromicro Conference on Software Engineering and Advanced Applications, SEAA 2020. - : Institute of Electrical and Electronics Engineers Inc.. - 9781728195322 ; , s. 276-283
-
Konferensbidrag (refereegranskat)abstract
- Researcher Bias (RB) occurs when researchers influence the results of an empirical study based on their expectations. RB might be due to the use of Questionable Research Practices (QRPs). In research fields like medicine, blinding techniques have been applied to counteract RB. We conducted an explorative qualitative survey to investigate RB in Software Engineering (SE) experiments, with respect to: (i) QRPs potentially leading to RB, (ii) causes behind RB, and (iii) possible actions to counteract RB including blinding techniques. Data collection was based on semi-structured interviews. We interviewed nine active experts in the empirical SE community. We then analyzed the transcripts of these interviews through thematic analysis. We found that some QRPs are acceptable in certain cases. Also, it appears that the presence of RB is perceived in SE and, to counteract RB, a number of solutions have been highlighted: some are intended for SE researchers and others for the boards of SE research outlets. © 2020 IEEE.
|
|
| 7. |
- Santos, Adrian, et al.
(författare)
-
A family of experiments on test-driven development
- 2021
-
Ingår i: Empirical Software Engineering. - : Springer. - 1382-3256 .- 1573-7616. ; 26:3
-
Tidskriftsartikel (refereegranskat)abstract
- Context:: Test-driven development (TDD) is an agile software development approach that has been widely claimed to improve software quality. However, the extent to which TDD improves quality appears to be largely dependent upon the characteristics of the study in which it is evaluated (e.g., the research method, participant type, programming environment, etc.). The particularities of each study make the aggregation of results untenable. Objectives:: The goal of this paper is to: increase the accuracy and generalizability of the results achieved in isolated experiments on TDD, provide joint conclusions on the performance of TDD across different industrial and academic settings, and assess the extent to which the characteristics of the experiments affect the quality-related performance of TDD. Method:: We conduct a family of 12 experiments on TDD in academia and industry. We aggregate their results by means of meta-analysis. We perform exploratory analyses to identify variables impacting the quality-related performance of TDD. Results:: TDD novices achieve a slightly higher code quality with iterative test-last development (i.e., ITL, the reverse approach of TDD) than with TDD. The task being developed largely determines quality. The programming environment, the order in which TDD and ITL are applied, or the learning effects from one development approach to another do not appear to affect quality. The quality-related performance of professionals using TDD drops more than for students. We hypothesize that this may be due to their being more resistant to change and potentially less motivated than students. Conclusion:: Previous studies seem to provide conflicting results on TDD performance (i.e., positive vs. negative, respectively). We hypothesize that these conflicting results may be due to different study durations, experiment participants being unfamiliar with the TDD process, or case studies comparing the performance achieved by TDD vs. the control approach (e.g., the waterfall model), each applied to develop a different system. Further experiments with TDD experts are needed to validate these hypotheses. © 2021, The Author(s), under exclusive licence to Springer Science+Business Media, LLC part of Springer Nature.
|
|
| 8. |
- Strandberg, Per Erik
(författare)
-
Automated System-Level Software Testing of Industrial Networked Embedded Systems
- 2021
-
Doktorsavhandling (övrigt vetenskapligt/konstnärligt)abstract
- Embedded systems are ubiquitous and play critical roles in management systems for industry and transport. Software failures in these domains may lead to loss of production or even loss of life, so the software in these systems needs to be reliable. Software testing is a standard approach for quality assurance of embedded software, and many software development processes strive for test automation. Out of the many challenges for successful software test automation, this thesis addresses five: (i) understanding how updated software reaches a test environment, how testing is conducted in the test environment, and how test results reach the developers that updated the software in the first place; (ii) selecting which test cases to execute in a test suite given constraints on available time and test systems; (iii) given that the test cases an run on different configurations of connected devices, selecting which hardware to use for each test case to be executed; (iv) analyzing test cases that, when executed over time on evolving software, testware or hardware revisions, appear to randomly fail; and (v) making test results information actionable with test results exploration and visualization.The challenges are tackled in several ways. First, to better understand the flow of information in the embedded systems software development process, interviews at five different companies were conducted. The results show how visualizations and a test results database support decision-making. Results also describe the overall flow of information in software testing: from developers to hardware in the test environment, and back to developers. Second, in order to address the challenges of test selection and hardware selection, automated approaches for testing given resource constraints were implemented and evaluated using industrial data stemming from years of nightly testing. It was shown that these approaches could solve problems such as nightly testing not finishing on time, as well as increasing hardware coverage by varying hardware selection over test iterations. Third, the challenge of intermittently failing tests was addressed with a new metric that can classify test cases as intermittently or consistently failing. Again, by using industry data, factors that lead to intermittent failures were identified, and similarities and differences between root causes for intermittently and consistently failing tests were observed. Finally, in order to better render test results actionable, a tool was implemented for test results exploration and visualization. The implementation was evaluated using a reference group and logging of the tool’s usage. Solution patterns and views of the tool were identified, as well as challenges for implementing such a tool.
|
|
| 9. |
- Turhan, Burak, et al.
(författare)
-
A Comparison of Cross- versus Single-company Effort Prediction Models for Web Projects
- 2014
-
Ingår i: 2014 40TH EUROMICRO CONFERENCE SERIES ON SOFTWARE ENGINEERING AND ADVANCED APPLICATIONS (SEAA 2014). - 9781479957958 ; , s. 285-292
-
Konferensbidrag (refereegranskat)abstract
- Background: In order to address the challenges in companies having no or limited effort datasets of their own, cross-company models have been a focus of interest for previous studies. Further, a particular domain of investigation has been Web projects. Aim: This study investigates to what extent effort predictions obtained using cross-company (CC) datasets are effective in relation to the predictions obtained using single-company (SC) datasets within the domain of web projects. Method: This study uses the Tukutuku database. We employed data on 125 projects from eight different companies and built cross and single-company models with stepwise linear regression (SWR) with and without relevancy filtering. We also benchmarked these models against mean and median based models. We report a case-by-case analysis per company as well as a meta-analysis of the findings. Results: Results showed that CC models provided poor predictions and performed significantly worse than SC models. However, relevancy filtered CC models yielded comparable results to that of SC models. These results corroborate with previous research. An interesting result was that the median-based models were consistently better than other models. Conclusions: We conclude that companies that carry out Web development may use a median-based CC model for prediction until it is possible for the company to build its own SC model, which can be used by itself or in combination with median-based estimations.
|
|
| 10. |
- Tuzun, Eray, et al.
(författare)
-
Ground-Truth Deficiencies in Software Engineering : When Codifying the Past Can Be Counterproductive
- 2022
-
Ingår i: IEEE Software. - : IEEE Computer Society. - 0740-7459 .- 1937-4194. ; 39:3, s. 85-95
-
Tidskriftsartikel (refereegranskat)abstract
- In software engineering, the objective function of human decision makers might be influenced by many factors. Relying on historical data as the ground truth may give rise to systems that automate software engineering decisions by mimicking past suboptimal behavior. We describe the problem and offer some strategies. ©IEEE.
|
|
