| 1. |
- Berntsson Svensson, Richard, 1978, et al.
(author)
-
Quality Requirements in Industrial Practice - An Extended Interview Study at Eleven Companies
- 2011
-
In: IEEE Transactions on Software Engineering. - 0098-5589 .- 1939-3520.
-
Journal article (peer-reviewed)abstract
- In order to create a successful software product and assure its quality, it is not enough to fulfill the functional requirements, it is also crucial to find the right balance among competing quality requirements (QR). An extended, previously piloted, interview study was performed to identify specific challenges associated with the selection, trade-off, and management of QR in industrial practice. Data was collected through semi-structured interviews with eleven product managers and eleven project leaders from eleven software companies. The contribution of this study is fourfold: First, it compares how QR are handled in two cases, companies working in business-to-business markets, and companies that are working in business-to-consumer markets. These two are also compared in terms of impact on the handling of QRs. Second, it compares the perceptions and priorities of QR by product and project management respectively. Third, it includes an examination of the interdependencies among quality requirements perceived as most important by the practitioners. Fourth, it characterizes the selection and management of QR in down-stream development activities.
|
|
| 2. |
- de Oliveira Neto, Francisco Gomes, et al.
(author)
-
Searching for models to evaluate software technology
- 2013
-
In: 2013 1st International Workshop on Combining Modelling and Search-Based Software Engineering, CMSBSE 2013 - Proceedings. - 9781467362849 ; , s. 12-15
-
Conference paper (peer-reviewed)abstract
- Modeling and abstraction is key in all engineering processes and have found extensive use also in software engineering. When developing new methodologies and techniques to support software engineers we want to evaluate them on realistic models. However, this is a challenge since (1) it is hard to get industry to give access to their models, and (2) we need a large number of models to systematically evaluate a technology. This paper proposes that search-based techniques can be used to search for models with desirable properties, which can then be used to systematically evaluate model-based technologies. By targeting properties seen in industrial models we can then get the best of both worlds: models that are similar to models used in industry but in quantities that allow extensive experimentation. To exemplify our ideas we consider a specific case in which a model generator is used to create models to test a regression test optimization technique. © 2013 IEEE.
|
|
| 3. |
- Engström, Emelie, et al.
(author)
-
Indirect effects in evidential assessment: A case study on regression test technology adoption
- 2012
-
In: 2nd International Workshop on Evidential Assessment of Software Technologies, EAST 2012. Lund, 22 September 2012. - New York, NY, USA : ACM. - 9781450315098 ; , s. 15-20
-
Conference paper (peer-reviewed)abstract
- Background: There is a need for effcient regression testing in most software development organizations. Often the proposed solutions involve automation. However, despite this being a well researched area, research results are rarely applied in industrial practice. Aim: In this paper we aim to bridge the gap between research and practice by providing examples of how evidence-based regression testing approaches can be adopted in industry. We also discuss challenges for the research community. Method: An industrial case study was carried out to evaluate the possibility to improve regression testing at Sony Ericsson Mobile Communications. We analyse the procedure undertaken based on frameworks from the evidence based software engineering, EBSE, paradigm (with a focus on the evidence) and automation literature (with a focus on the practical effects). Results: Our results pinpoint the need for systematic approaches when introducing a new tool. Practitioners and researchers need congruent guidelines supporting the appraisal of both the evidence base and the pragmatic effects, both direct but also indirect, of the changes. This is illustrated by the introduction of the automation perspective.
|
|
| 4. |
- Ghazi, Ahmad Nauman, et al.
(author)
-
Information sources and their importance to prioritize test cases in heterogeneous systems context
- 2014
-
In: Communications in Computer and Information Science. - Berlin, Heidelberg : Springer. - 1865-0929. - 9783662438954 - 9783662438961 ; 425, s. 86-98
-
Conference paper (peer-reviewed)abstract
- Context: Testing techniques proposed in the literature rely on various sources of information for test case selection (e.g., require- ments, source code, system structure, etc.). The challenge of test selection is amplified in the context of heterogeneous systems, where it is unknown which information/data sources are most important. Contribution: (1) Achieve in-depth understanding of test processes in heterogeneous systems; (2) Elicit information sources for test selection in the context of heterogeneous systems. (3) Capture the relative importance of the identified information sources. Method: Case study research is used for the elicitation and understanding of which information sources are relevant for test case privatization, followed by an exploratory survey capturing the relative importance of information sources for testing heterogeneous systems. Results: We classified different information sources that play a vital role in the test selection process, and found that their importance differs largely for the different test levels observed in heterogeneous testing. However, overall all sources were considered essential in test selection for heterogeneous systems. Conclusion: Heterogeneous system testing requires solutions that take all information sources into account when suggesting test cases for selection. Such approaches need to be developed and compared with existing solutions.
|
|
| 5. |
- Gren, Lucas, 1984, et al.
(author)
-
Work Motivational Challenges Regarding the Interface Between Agile Teams and a Non-Agile Surrounding Organization: A case study
- 2014
-
In: Agile Conference (AGILE), 2014. July 28 2014-Aug.. - : IEEE Press. - 9780769552224 ; , s. 11-15
-
Conference paper (peer-reviewed)abstract
- There are studies showing what happens if agile teams are introduced into a non-agile organization, e.g. higher overhead costs and the necessity of an understanding of agile methods even outside the teams. This case study shows an example of work motivational aspects that might surface when an agile team exists in the middle of a more traditional structure. This case study was conducted at a car manufacturer in Sweden, consisting of an unstructured interview with the Scrum Master and a semi-structured focus group. The results show that the teams felt that the feedback from the surrounding organization was unsynchronized resulting in them not feeling appreciated when delivering their work. Moreover, they felt frustrated when working on non-agile teams after have been working on agile ones. This study concludes that there were work motivational affects of fitting an agile team into a non-agile surrounding organization, and therefore this might also be true for other organizations.
|
|
| 6. |
- Holt, Nina E., et al.
(author)
-
Empirical evaluations on the cost-effectiveness of state-based testing: An industrial case study
- 2014
-
In: Information and Software Technology. - : Elsevier BV. - 0950-5849 .- 1873-6025. ; 56:8, s. 890-910
-
Journal article (peer-reviewed)abstract
- Context: Test models describe the expected behavior of the software under test and provide the basis for test case and oracle generation. When test models are expressed as UML state machines, this is typically referred to as state-based testing (SBT). Despite the importance of being systematic while testing, all testing activities are limited by resource constraints. Thus, reducing the cost of testing while ensuring sufficient fault detection is a common goal in software development. No rigorous industrial case studies of SBT have yet been published. Objective: In this paper, we evaluate the cost-effectiveness of SBT on actual control software by studying the combined influence of four testing aspects: coverage criterion, test oracle, test model and unspecified behavior (sneak paths). Method: An industrial case study was used to investigate the cost-effectiveness of SBT. To enable the evaluation of SBT techniques, a model-based testing tool was configured and used to automatically generate test suites. The test suites were evaluated using 26 real faults collected in a field study. Results: Results show that the more detailed and rigorous the test model and oracle, the higher the fault-detection ability of SBT. A less precise oracle achieved 67% fault detection, but the overall cost reduction of 13% was not enough to make the loss an acceptable trade-off. Removing details from the test model significantly reduced the cost by 85%. Interestingly, only a 24–37% reduction in fault detection was observed. Testing for sneak paths killed the remaining eleven mutants that could not be killed by the conformance test strategies. Conclusions: Each of the studied testing aspects influences cost-effectiveness and must be carefully considered in context when selecting strategies. Regardless of these choices, sneak-path testing is a neces- sary step in SBT since sneak paths are common while also undetectable by conformance testing.
|
|
| 7. |
- Lindqvist, Erik, et al.
(author)
-
Outliers and Replication in Software Engineering
- 2014
-
In: Proceedings - Asia-Pacific Software Engineering Conference, APSEC. - 1530-1362. - 9781479974252 ; 1, s. 207-214
-
Conference paper (peer-reviewed)abstract
- Empirical software engineering is a research field of growing interest. Studies within this field handles an increasing amount of data. In order to replicate a study the data needs to be accessible and all processing of this data needs to be reproducible. Specifically, the handling of deviating data points, also known as outliers, needs to be documented in order for a study to be replicated. This study investigated the data availability for recently published studies within empirical software engineering. Furthermore, it also investigated if outliers are documented in the same research field. Papers were reviewed using a literature review and the presence of outliers was investigated using an unsupervised outlier detection method. Only 37% of the papers reviewed had their data accessible. Furthermore, in many cases outliers were present in the reviewed studies but 63% of the papers studies did not mention how outliers were handled. The data availability within empirical software engineering research is low and is hindering replication of studies. Additionally, the lack of documentation regarding how outliers are handled is hindering replication.
|
|
| 8. |
|
|
| 9. |
- Madeyski, L., et al.
(author)
-
Overcoming the Equivalent Mutant Problem: A Systematic Literature Review and a Comparative Experiment of Second Order Mutation
- 2014
-
In: Ieee Transactions on Software Engineering. - : Institute of Electrical and Electronics Engineers (IEEE). - 0098-5589 .- 1939-3520. ; 40:1, s. 23-42
-
Journal article (peer-reviewed)abstract
- Context. The equivalent mutant problem (EMP) is one of the crucial problems in mutation testing widely studied over decades. Objectives. The objectives are: to present a systematic literature review (SLR) in the field of EMP; to identify, classify and improve the existing, or implement new, methods which try to overcome EMP and evaluate them. Method. We performed SLR based on the search of digital libraries. We implemented four second order mutation (SOM) strategies, in addition to first order mutation (FOM), and compared them from different perspectives. Results. Our SLR identified 17 relevant techniques (in 22 articles) and three categories of techniques: detecting (DEM); suggesting (SEM); and avoiding equivalent mutant generation (AEMG). The experiment indicated that SOM in general and JudyDiffOp strategy in particular provide the best results in the following areas: total number of mutants generated; the association between the type of mutation strategy and whether the generated mutants were equivalent or not; the number of not killed mutants; mutation testing time; time needed for manual classification. Conclusions. The results in the DEM category are still far from perfect. Thus, the SEM and AEMG categories have been developed. The JudyDiffOp algorithm achieved good results in many areas.
|
|
| 10. |
- Mairhofer, Stefan, et al.
(author)
-
Search-based Software Testing and Test Data Generation for a Dynamic Programming Language
- 2011
-
In: Proceedings of the Genetic and Evolutionary Computation Conference (GECCO).
-
Conference paper (peer-reviewed)abstract
- Manually creating test cases is time consuming and error prone. Search-based software testing can help automate this process and thus reduce time and effort and increase quality by automatically generating relevant test cases. Previous research has mainly focused on static programming languages and simple test data inputs such as numbers. This is not practical for dynamic programming languages that are increasingly used by software developers. Here we present an approach for search-based software testing for dynamically typed programming languages that can generate test scenarios and both simple and more complex test data. The approach is implemented as a tool, RuTeG, in and for the dynamic programming language Ruby. It combines an evolutionary search for test cases that give structural code coverage with a learning component to restrict the space of possible types of inputs. The latter is called for in dynamic languages since we cannot always know statically which types of objects are valid inputs. Experiments on 14 cases taken from real-world Ruby projects show that RuTeG achieves full or higher statement coverage on more cases and does so faster than randomly generated test cases.
|
|
| 11. |
- Marculescu, Bogdan, 1984, et al.
(author)
-
Objective re-weighting to guide an interactive search based software testing system
- 2013
-
In: 2013 12th International Conference on Machine Learning and Applications, ICMLA 2013; Miami, FL; United States; 4 December 2013 through 7 December 2013. - Miami : IEEE. ; 2, s. 102-107
-
Conference paper (peer-reviewed)abstract
- Even hardware-focused industries today develop products where software is both a large and important component. Engineers tasked with developing and integrating these products do not always have a software engineering background. To ensure quality, tools are needed that automate and support software testing while allowing these domain specialists to leverage their knowledge and experience. Search-based testing could be a key aspect in creating an automated tool for supporting testing activities. However, domain specific quality criteria and trade-offs make it difficult to develop a general fitness function a priori, so interaction between domain specialists and such a tool would be critical to its success. In this paper we present a system for interactive search based software testing and investigate a way for domain specialists to guide the search by dynamically re-weighting quality goals. Our empirical investigation shows that objective reweighing can help a human domain specialist interactively guide the search, without requiring specialised knowledge of the system and without sacrificing population diversity.
|
|
| 12. |
- Radjenovic, D., et al.
(author)
-
Software fault prediction metrics: A systematic literature review
- 2013
-
In: Information and Software Technology. - : Elsevier BV. - 0950-5849 .- 1873-6025. ; 55:8, s. 1397-1418
-
Journal article (peer-reviewed)abstract
- Context: Software metrics may be used in fault prediction models to improve software quality by predicting fault location. Objective: This paper aims to identify software metrics and to assess their applicability in software fault prediction. We investigated the influence of context on metrics' selection and performance. Method: This systematic literature review includes 106 papers published between 1991 and 2011. The selected papers are classified according to metrics and context properties. Results: Object-oriented metrics (49%) were used nearly twice as often compared to traditional source code metrics (27%) or process metrics (24%). Chidamber and Kemerer's (CK) object-oriented metrics were most frequently used. According to the selected studies there are significant differences between the metrics used in fault prediction performance. Object-oriented and process metrics have been reported to be more successful in finding faults compared to traditional size and complexity metrics. Process metrics seem to be better at predicting post-release faults compared to any static code metrics. Conclusion: More studies should be performed on large industrial software systems to find metrics more relevant for the industry and to answer the question as to which metrics should be used in a given context.
|
|
| 13. |
- Rogstad, Erik, et al.
(author)
-
Test case selection for black-box regression testing of database applications
- 2013
-
In: Information and Software Technology. - : Elsevier BV. - 0950-5849 .- 1873-6025. ; 55:10, s. 1781-1795
-
Journal article (peer-reviewed)abstract
- Context: This paper presents an approach for selecting regression test cases in the context of large-scale, database applications. We focus on a black-box (specification-based) approach, relying on classification tree models to model the input domain of the system under test (SUT), in order to obtain a more practical and scalable solution. We perform an industrial case study where the SUT is a large database application in Norway’s tax department. Objective: We investigate the use of similarity-based test case selection for supporting black box regression testing of database applications. We have developed a practical approach and tool (DART) for functional black-box regression testing of database applications. In order to make the regression test approach scalable for large database applications, we needed a test case selection strategy that reduces the test execution costs and analysis effort. We used classification tree models to partition the input domain of the SUT in order to then select test cases. Rather than selecting test cases at random from each partition, we incorporated a similarity-based test case selection, hypothesizing that it would yield a higher fault detection rate. Method: An experiment was conducted to determine which similarity-based selection algorithm was the most suitable in selecting test cases in large regression test suites, and whether similarity-based selection was a worthwhile and practical alternative to simpler solutions. Results: The results show that combining similarity measurement with partition-based test case selection, by using similarity-based test case selection within each partition, can provide improved fault detection rates over simpler solutions when specific conditions are met regarding the partitions. Conclusions: Under the conditions present in the experiment the improvements were marginal. However, a detailed analysis concludes that the similarity-based selection strategy should be applied when a large number of test cases are contained in each partition and there is significant variability within partitions. If these conditions are not present, incorporating similarity measures is not worthwhile, since the gain is negligible over a random selection within each partition.
|
|
| 14. |
- Schneider, S., et al.
(author)
-
Solutions in global software engineering: A systematic literature review
- 2013
-
In: International Journal of Information Management. - : Elsevier BV. - 0268-4012 .- 1873-4707. ; 33:1, s. 119-132
-
Journal article (peer-reviewed)abstract
- Global software engineering (GSE) has received increased attention, as globalization enables and encourages increased distribution of product development. Many empirical studies and systematic literature reviews (SLRs) focus on the identification of challenges, this paper however presents the first SLR collecting and analyzing solutions associated with GSE, while also evaluating the level of empirical validation of said solutions. As a starting point the paper presents a GSE model, designed to categorize solutions into process areas, useful for the analysis of the research community's contributions to state-of-the-art and identifying fundamental gaps in research. In addition, the model categorizing the solutions is populated with references and good-examples, useful for practitioners, which can use the model to find solutions to overall challenges in various process areas. The overall results of the systematic review revealed more than 330 papers containing 127 solutions that were then identified and mapped to the model. The process areas of project management are highly populated, while other areas like product integration have received surprisingly little attention. In addition, selected process area is elaborated upon in terms of contents and deficiencies.
|
|
| 15. |
- Svensson, R. B., et al.
(author)
-
Quality Requirements in Industrial Practice-An Extended Interview Study at Eleven Companies
- 2012
-
In: IEEE Transactions on Software Engineering. - : Institute of Electrical and Electronics Engineers (IEEE). - 0098-5589 .- 1939-3520. ; 38:4, s. 923-935
-
Journal article (peer-reviewed)abstract
- In order to create a successful software product and assure its quality, it is not enough to fulfill the functional requirements, it is also crucial to find the right balance among competing quality requirements (QR). An extended, previously piloted, interview study was performed to identify specific challenges associated with the selection, tradeoff, and management of QR in industrial practice. Data were collected through semistructured interviews with 11 product managers and 11 project leaders from 11 software companies. The contribution of this study is fourfold: First, it compares how QR are handled in two cases, companies working in business-to-business markets and companies that are working in business-to-consumer markets. These two are also compared in terms of impact on the handling of QR. Second, it compares the perceptions and priorities of QR by product and project management, respectively. Third, it includes an examination of the interdependencies among quality requirements perceived as most important by the practitioners. Fourth, it characterizes the selection and management of QR in downstream development activities.
|
|
| 16. |
- Torkar, Richard, 1971, et al.
(author)
-
Prediction of faults-slip-through in large software projects: An empirical evaluation
- 2014
-
In: Software quality journal. - Netherlands : Springer Science and Business Media LLC. - 0963-9314 .- 1573-1367. ; 22:1, s. 51-86
-
Journal article (peer-reviewed)abstract
- A large percentage of the cost of rework can be avoided by finding more faults earlier in a software test process. Therefore, determination of which software test phases to focus improvement work on has considerable industrial interest. We evaluate a number of prediction techniques for predicting the number of faults slipping through to unit, function, integration, and system test phases of a large industrial project. The objective is to quantify improvement potential in different test phases by striving toward finding the faults in the right phase. The results show that a range of techniques are found to be useful in predicting the number of faults slipping through to the four test phases; however, the group of search-based techniques (genetic programming, gene expression programming, artificial immune recognition system, and particle swarm optimization–based artificial neural network) consistently give better predictions, having a representation at all of the test phases. Human predictions are consistently better at two of the four test phases. We conclude that the human predictions regarding the number of faults slipping through to various test phases can be well supported by the use of search-based techniques. A combination of human and an automated search mechanism (such as any of the search-based techniques) has the potential to provide improved prediction results.
|
|
| 17. |
- Torkar, Richard, 1971
(author)
-
State-based testing: Industrial evaluation of the cost effectiveness of round-trip path and sneak-path testing strategies
- 2012
-
In: International Symposium on Software Reliability Engineering (ISSRE).
-
Conference paper (peer-reviewed)abstract
- In the context of safety-critical software development, one important step in ensuring safe behavior is conformance testing, i.e., checking compliance between expected behavior and implementation. Round-trip path testing (RTP) is one example of conformance testing. Another essential step, however, is sneak- path testing, that is testing of how software reacts to unexpected events for a particular system state. Despite the importance of being systematic while testing, all testing activities take place, even for safety-critical software, under resource constraints. In this paper, we present an empirical evaluation of the cost- effectiveness of RTP when combined with sneak-path testing in the context of an industrial control system. Results highlight the importance of sneak-path testing since unexpected behavior is shown to be difficult to detect by other common, state-based test strategies. Results also suggest that sneak-path testing is a cost- effective supplement to RTP.
|
|
