| 1. |
- Bell, R, et al.
(författare)
-
The limited impact of individual developer data on software defect prediction
- 2013
-
Ingår i: Empirical Software Engineering. - : Springer Science and Business Media LLC. - 1382-3256 .- 1573-7616. ; 18:3, s. 478-505
-
Tidskriftsartikel (refereegranskat)abstract
- Previous research has provided evidence that a combination of static code metrics and software history metrics can be used to predict with surprising success which files in the next release of a large system will havethe largest numbers of defects. In contrast, very little research exists to indicate whether information about individual developers can profitably be used to improve predictions. We investigate whether files in a large system that are modified by an individual developer consistently contain either more or fewer faults than the average of all files in the system. The goal of the investigation is to determine whether information about which particular developer modified a file is able to improve defect predictions. We also extend earlier research evaluating use of counts of the number of developers who modified a file as predictors of the file's future faultiness. We analyze change reports filed for three large systems, each containing 18 releases, with a combined total of nearly 4 million LOC and over 11,000 files. A buggy file ratio is defined for programmers, measuring the proportion of faulty files in Release R out of all files modified by the programmer in Release R-1. We assess the consistency of the buggy file ratio across releases for individual programmers both visually and within the context of a fault prediction model. Buggy file ratios for individual programmers often varied widely across all the releases that they participated in. A prediction model that takes account of the history of faulty files that were changed by individual developers shows improvement over the standard negative binomial model of less than 0.13% according to one measure, and no improvement at all according to another measure. In contrast, augmenting a standard model with counts of cumulative developers changing files in prior releases produced up to a 2% improvement in the percentage of faults detected in the top 20% of predicted faulty files. The cumulative number of developers interacting with a file can be a useful variable for defect prediction. However, the study indicates that adding information to a model about which particular developermodified a file is not likely to improve defect predictions.
|
|
| 2. |
- Derehag, Jesper, et al.
(författare)
-
Transitioning Fault Prediction Models to a New Environment
- 2016
-
Ingår i: Proceedings - 2016 12th European Dependable Computing Conference, EDCC 2016. - 9781509015825 ; , s. 241-248
-
Konferensbidrag (refereegranskat)abstract
- We describe the application and evaluation of fault prediction algorithms to a project developed by a Swedish company that transitioned from waterfall to agile development methods. The project used two different version control systems and a separate bug tracking system during its lifetime. The algorithms were originally designed for use on systems implemented with a traditional waterfall process at American companies that maintained their project records in an integrated database system that combined bug recording and version control. We compare the performance of the original prediction model on the American systems to the results obtained in the Swedish environment in both its pre-agile and agile stages. We also consider the impact of additional variables in the model.
|
|
| 3. |
- Enoiu, Eduard Paul, et al.
(författare)
-
Automated Test Generation using Model-Checking: An Industrial Evaluation
- 2016
-
Ingår i: International Journal on Software Tools for Technology Transfer. - Germany : Springer. - 1433-2779 .- 1433-2787. ; 18:3, s. 335-353
-
Tidskriftsartikel (refereegranskat)abstract
- In software development, testers often focus on functional testing to validate implemented programs against their specifications. In safety critical software development, testers are also required to show that tests exercise, or cover, the structure and logic of the implementation. To achieve different types of logic coverage, various program artifacts such as decisions and conditions are required to be exercised during testing. Use of model-checking for structural test generation has been proposed by several researchers. The limited application to models used in practice and the state-space explosion can, however, impact model-checking and hence the process of deriving tests for logic coverage. Thus, there is a need to validate these approaches against relevant industrial systems such that more knowledge is built on how to efficiently use them in practice. In this paper, we present a tool-supported approach to handle software written in the Function Block Diagram language such that logic coverage criteria can be formalized and used by a model-checker to automatically generate tests. To this end, we conducted a study based on industrial use-case scenarios from Bombardier Transportation AB, showing how our toolbox COMPLETETEST can be applied to generate tests in software systems used in the safety-critical domain. To evaluate the approach, we applied the toolbox to 157 programs and found that it is efficient in terms of time required to generate tests that satisfy logic coverage and scales well for most of the programs.
|
|
| 4. |
- Strandberg, Per, et al.
(författare)
-
Automated System Level Regression Test Prioritization in a Nutshell
- 2017
-
Ingår i: IEEE Software. - 0740-7459 .- 1937-4194. ; 34:4, s. 30-37
-
Tidskriftsartikel (refereegranskat)abstract
- Westermo Research and Development has developed SuiteBuilder, an automated tool to determine an effective ordering of regression test cases. The ordering is based on factors such as fault detection success, the interval since the last execution, and code modifications. SuiteBuilder has enabled Westermo to overcome numerous regression-testing problems, including lack of time to run a complete regression suite, failure to detect bugs in a timely manner, and repeatedly omitted tests. In the tool's first two years of use, reordered test suites finished in the available time, most fault-detecting test cases were located in the first third of suites, no important test case was omitted, and the necessity for manual work on the suites decreased greatly.
|
|
| 5. |
- Strandberg, Per Erik, et al.
(författare)
-
Automated test mapping and coverage for network topologies
- 2018
-
Ingår i: ISSTA 2018 - Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis. - New York, NY, USA : Association for Computing Machinery, Inc. - 9781450356992 ; , s. 73-83
-
Konferensbidrag (refereegranskat)abstract
- Communication devices such as routers and switches play a critical role in the reliable functioning of embedded system networks. Dozens of such devices may be part of an embedded system network, and they need to be tested in conjunction with various computational elements on actual hardware, in many different configurations that are representative of actual operating networks. An individual physical network topology can be used as the basis for a test system that can execute many test cases, by identifying the part of the physical network topology that corresponds to the configuration required by each individual test case. Given a set of available test systems and a large number of test cases, the problem is to determine for each test case, which of the test systems are suitable for executing the test case, and to provide the mapping that associates the test case elements (the logical network topology) with the appropriate elements of the test system (the physical network topology). We studied a real industrial environment where this problem was originally handled by a simple software procedure that was very slow in many cases, and also failed to provide thorough coverage of each network's elements. In this paper, we represent both the test systems and the test cases as graphs, and develop a new prototype algorithm that a) determines whether or not a test case can be mapped to a subgraph of the test system, b) rapidly finds mappings that do exist, and c) exercises diverse sets of network nodes when multiple mappings exist for the test case. The prototype has been implemented and applied to over 10,000 combinations of test cases and test systems, and reduced the computation time by a factor of more than 80 from the original procedure. In addition, relative to a meaningful measure of network topology coverage, the mappings achieved an increased level of thoroughness in exercising the elements of each test system.
|
|
| 6. |
- Strandberg, Per Erik, et al.
(författare)
-
Experience Report : Automated System Level Regression Test Prioritization Using Multiple Factors
- 2016
-
Ingår i: 27th International Symposium on Software Reliability Engineering ISSRE'16.
-
Konferensbidrag (refereegranskat)abstract
- We propose a new method of determining an effective ordering of regression test cases, and describe its implementation as an automated tool called SuiteBuilder developed by Westermo Research and Development AB. The tool generates an efficient order to run the cases in an existing test suite by using expected or observed test duration and combining priorities of multiple factors associated with test cases, including previous fault detection success, interval since last executed, and modifications to the code tested. The method and tool were developed to address problems in the traditional process of regression testing, such as lack of time to run a complete regression suite, failure to detect bugs in time, and tests that are repeatedly omitted. The tool has been integrated into the existing nightly test framework for Westermo software that runs on large-scale data communication systems. In experimental evaluation of the tool, we found significant improvement in regression testing results. The re-ordered test suites finish within the available time, the majority of fault-detecting test cases are located in the first third of the suite, no important test case is omitted, and the necessity for manual work on the suites is greatly reduced.
|
|
| 7. |
- Strandberg, Per Erik, et al.
(författare)
-
Intermittently Failing Tests in the Embedded Systems Domain
- 2020
-
Ingår i: International Symposium on Software Testing and Analysis ISSTA'20. - New York, NY, USA : ACM. ; , s. 337-348
-
Konferensbidrag (refereegranskat)abstract
- Software testing is sometimes plagued with intermittently failing tests and finding the root causes of such failing tests is often difficult. This problem has been widely studied at the unit testing level for open source software, but there has been far less investigation at the system test level, particularly the testing of industrial embedded systems. This paper describes our investigation of the root causes of intermittently failing tests in the embedded systems domain, with the goal of better understanding, explaining and categorizing the underlying faults. The subject of our investigation is a currently-running industrial embedded system, along with the system level testing that was performed. We devised and used a novel metric for classifying test cases as intermittent. From more than a half million test verdicts, we identified intermittently and consistently failing tests, and identified their root causes using multiple sources. We found that about 1-3% of all test cases were intermittently failing. From analysis of the case study results and related work, we identified nine factors associated with test case intermittence. We found that a fix for a consistently failing test typically removed a larger number of failures detected by other tests than a fix for an intermittent test. We also found that more effort was usually needed to identify fixes for intermittent tests than for consistent tests. An overlap between root causes leading to intermittent and consistent tests was identified. Many root causes of intermittence are the same in industrial embedded systems and open source software. However, when comparing unit testing to system level testing, especially for embedded systems, we observed that the test environment itself is often the cause of intermittence.
|
|
| 8. |
- Weyuker, Elaine, et al.
(författare)
-
Experiences with academic-industrial collaboration on empirical studies of software systems
- 2017
-
Ingår i: 2017 IEEE 28TH INTERNATIONAL SYMPOSIUM ON SOFTWARE RELIABILITY ENGINEERING WORKSHOPS (ISSREW 2017). - : Institute of Electrical and Electronics Engineers Inc.. ; , s. 164-168
-
Konferensbidrag (refereegranskat)abstract
- The authors have held both academic and industrial research positions, and have designed and carried out many empirical studies of large software systems that were built and maintained in industrial environments. Their experiences show that the most crucial component of a successful study is the participation of at least one industrial collaborator who is committed to the study’s goals and is able to provide advice and assistance throughout the course of the study. This paper describes studies carried out in three different industrial environments, discusses obstacles that arise, and how the authors have been able to overcome some of those obstacles.
|
|
| 9. |
|
|
