| 1. |
|
|
| 2. |
- Causevic, Adnan, et al.
(författare)
-
An Industrial Survey on Contemporary Aspects of Software Testing
- 2010
-
Ingår i: ICST 2010 - 3rd International Conference on Software Testing, Verification and Validation. - Paris, France. - 9780769539904 ; , s. 393-401
-
Konferensbidrag (refereegranskat)abstract
- Software testing is a major source of expense in software projects and a proper testing process is a critical ingredient in the cost-efficient development of high-quality software. Contemporary aspects, such as the introduction of a more lightweight process, trends towards distributed development, and the rapid increase of software in embedded and safety-critical systems, challenge the testing process in unexpected manners. To our knowledge, there are very few studies focusing on these aspects in relation to testing as perceived by different contributors in the software development process. This paper qualitatively and quantitatively analyses data from an industrial questionnaire survey, with a focus on current practices and preferences on contemporary aspects of software testing. Specifically, the analysis focuses on perceptions of the software testing process in different categories of respondents. Categorization of respondents is based on safety-criticality, agility, distribution of development, and application domain. While confirming some of the commonly acknowledged facts, our findings also reveal notable discrepancies between preferred and actual testing practices. We believe continued research efforts are essential to provide guidelines in the adaptation of the testing process to take care of these discrepancies, thus improving the quality and efficiency of the software development.
|
|
| 3. |
- Causevic, Adnan, et al.
(författare)
-
Effects of Negative Testing on TDD : An Industrial Experiment
- 2013
-
Ingår i: Agile Processes in Software Engineering and Extreme Programming. - Berlin, Heidelberg : Springer. - 9783642383137 - 9783642383144 ; , s. 91-105
-
Bokkapitel (refereegranskat)abstract
- In our recent academic experiments, an existence of positive test bias, that is lack of negative test cases, was identified when a test driven development approach was used. At the same time, when defect detecting ability of individual test cases was calculated, it was noted that the probability of a negative test case to detect a defect was substantially higher than that of a positive test case. The goal of this study is to investigate the existence of positive test bias in test driven development within an industrial context, and measure defect detecting ability of both positive and negative test cases. An industrial experiment was conducted at Infosys Ltd. India, whose employees voluntarily signed up to participate in the study and were randomly assigned to groups utilizing test driven development, test driven development with negative testing, and test last development. Source code and test cases created by each participant during the study were collected and analysed. The collected data indicate a statistically significant difference between the number of positive and negative test cases created by industrial participants, confirming the existence of positive test bias. The difference in defect detecting ability of positive and negative test cases is also statistically significant. As a result, similarly to our previous academic study, 29% of all test cases were negative, contributing by revealing as much as 71% of all the defects found by all test cases. With this industrial experiment, we confirmed the existence of a positive test bias in an industrial context, as well as significantly higher defect detecting ability of negative test cases.
|
|
| 4. |
|
|
| 5. |
- Causevic, Adnan, et al.
(författare)
-
Factors Limiting Industrial Adoption of Test Driven Development: A Systematic Review
- 2011
-
Ingår i: 4th IEEE International Conference on Software Testing, Verification, and Validation, ICST 2011; Berlin; 21 March 2011 through 25 March 2011. - 9780769543420 ; , s. 337-346
-
Konferensbidrag (refereegranskat)abstract
- Test driven development (TDD) is one of the basic practices of agile software development and both academia and practitioners claim that TDD, to a certain extent, improves the quality of the code produced by developers. However, recent results suggest that this practice is not followed to the extent preferred by industry. In order to pinpoint specific obstacles limiting its industrial adoption we have conducted a systematic literature review on empirical studies explicitly focusing on TDD as well as indirectly addressing TDD. Our review has identified seven limiting factors viz., increased development time, insufficient TDD experience/knowledge, lack of upfront design, domain and tool specific issues, lack of developer skill in writing test cases, insufficient adherence to TDD protocol, and legacy code. The results of this study is of special importance to the testing community, since it outlines the direction for further detailed scientific investigations as well as highlights the requirement of guidelines to overcome these limiting factors for successful industrial adoption of TDD.
|
|
| 6. |
- Causevic, Adnan, et al.
(författare)
-
Impact of Test Design Technique Knowledge on Test Driven Development: A Controlled Experiment
- 2012
-
Ingår i: Lecture Notes in Business Information Processing, vol. 111 LNBIP. - Berlin, Heidelberg : Springer. - 9783642303494 ; , s. 138-152
-
Bokkapitel (refereegranskat)abstract
- Agile development approaches are increasingly being followed and favored by the industry. Test Driven Development (TDD) is a key agile practice and recent research results suggest that the successful adoption of TDD depends on different limiting factors, one of them being insufficient developer testing skills. The goal of this paper is to investigate if developers who are educated on general testing knowledge will be able to utilize TDD more efectively. We conducted a controlled experiment with master students during the course on Software Verification & Validation (V&V) where source code and test cases created by each participant during the labs as well as their answers on a survey questionnaire were collected and analyzed. Descriptive statistics indicate improvements in statement coverage. However, no statistically significant differences could be established between the pre- and post-course groups of students. By qualitative analysis of students’ tests, we noticed a lack of test cases for non-stated requirements ("negative" tests) resulting in a non-detection of bugs. Students did show preference towards TDD in surveys. Although further research is required to fully establish this, we believe that identifying specific testing knowledge which is complementary to the testing skills of a new TDD developer would enable developers to perform their tasks in a more eficient manner.
|
|
| 7. |
|
|
| 8. |
- Causevic, Adnan, et al.
(författare)
-
Quality of Testing in Test Driven Development
- 2012
-
Ingår i: 2012 8th International Conference on the Quality of Information and Communications Technology, QUATIC 2012. - : IEEE. - 9780769547770 ; , s. 266-271
-
Konferensbidrag (refereegranskat)abstract
- Test-driven development is an essential part of eXtreme Programming approach with the preference of being followed in other Agile methods as well. For several years, researchers are performing empirical investigations to evaluate quality improvements in the resulting code when test-driven development is being used. However, very little had been reported into investigating the quality of the testing performed in conjunction with test-driven development. In this paper we present results from an experiment specifically designed to evaluate the quality of test cases created by developers who used the test-first and the traditional test-last approaches. On an average, the quality of testing in test-driven development was almost the same as the quality of testing using test-last approach. However, detailed analysis of test cases, created by test-driven development group, revealed that 29% of test cases were "negative" test cases (based on non-specified requirements) but contributing as much as 65% to the overall tests quality score of test-first developers. We are currently investigating the possibility of extending test-driven development to facilitate non-specified requirements to a higher extent and thus minimise the impact of a potentially inherent effect of positive test bias.
|
|
| 9. |
|
|
| 10. |
- Causevic, Adnan, et al.
(författare)
-
TDDHQ : Achieving Higher Quality Testing in Test Driven Development
- 2013
-
Ingår i: Euromicro Conference on Software Engineering and Advanced Applications. ; , s. 33-36
-
Konferensbidrag (refereegranskat)abstract
- Test driven development (TDD) appears not to be immune to positive test bias effects, as we observed in several empirical studies. In these studies, developers created a significantly larger set of positive tests, but at the same time the number of defects detected with negative tests is significantly higher than those detected by positive ones. In this paper we propose the concept of TDDHQ which is aimed at achieving higher quality of testing in TDD by augmenting the standard TDD with suitable test design techniques. To exemplify this concept, we present combining equivalence partitioning test design technique together with the TDD, for the purpose of improving design of test cases. Initial evaluation of this approach showed a noticeable improvement in the quality of test cases created by developers utilising TDDHQ approach.
|
|
| 11. |
|
|
| 12. |
- Causevic, Adnan, et al.
(författare)
-
Test Case Quality in Test Driven Development: A Study Design and a Pilot Experiment
- 2012
-
Ingår i: EASE 2012, Proceedings. - : IET. - 9781849195416 ; , s. 223-227
-
Konferensbidrag (refereegranskat)abstract
- Background: Test driven development, as a side-effect of developing software, will produce a set of accompanied test cases which can protect implemented features during code refactoring. However, recent research results point out that successful adoption of test driven development might be limited by the testing skills of developers using it. Aim: Main goal of this paper is to investigate if there is a difference between the quality of test cases created while using test-first and test-last approaches. Additional goal of this paper is to measure the code quality produced using test-first and test-last approaches. Method: A pilot study was conducted during the master level course on Software Verification & Validation at Mälardalen University. Students were working individually on the problem implementation by being randomly assigned to a test-first or a test-last (control) group. Source code and test cases created by each participant during the study, as well as their answers on a survey questionnaire after the study, were collected and analysed. The quality of the test cases is analysed from three perspectives: (i) code coverage, (ii) mutation score and (iii) the total number of failing assertions. Results: The total number of test cases with failing assertions (test cases revealing an error in the code) was nearly the same for both test-first and test-last groups. This can be interpreted as "test cases created by test-first developers were as good as (or as bad as) test cases created by test-last developers". On the contrary, solutions created by test-first developers had, on average, 27% less failing assertions when compared to solutions created by the test-last group. Conclusions: Though the study provided some interesting observations, it needs to be conducted as a fully controlled experiment with a higher number of participants in order to validate statistical significance of the presented results.
|
|
| 13. |
|
|
| 14. |
- Ekman, M., et al.
(författare)
-
Tool qualification for safety related systems
- 2014
-
Ingår i: Ada User Journal. - 1381-6551. ; 35, s. 47-54
-
Tidskriftsartikel (refereegranskat)abstract
- Tools used in the development of safety related software applications need to be qualified as safe. That is, the tools cannot be allowed to introduce hazardous faults into the application, e.g., a compiler shall not generate dangerous code due to failure of the compiler. In many cases laws and regulations require the product development of safety related applications to comply with industry sector specific safety standards. Examples of such standards include EN50129/50128 for railway applications, ISO/EN13849 for machines with moving parts, DO-178B/C for avionics, or IS026262 for cars. These standards require the use of a rigorous development and maintenance process. The standards are also mainly intended to be used when developing systems from scratch. However, most development and test tools are not developed from scratch according to the rigorous processes of these standards. In order to address this issue, some of the standards provide means for qualifying existing tools as a more lightweight and pragmatic alternative to a regular certification process. In this paper we analyze the concept of these qualification approaches. The result of the analysis in our contribution includes a set of approaches that can be applied individually or as a combination in order to reduce the effort needed for qualifying tools. As a running example we use one of the most flexible but at the same time dangerous, even prohibited, maintenance techniques available: dynamic instrumentation of executing code. With this example, we describe how exceptions in these standards can be utilized in order to qualify a dynamic instrumentation tool with a minimal effort, without following the process of tool certification as defined by the standards.
|
|
| 15. |
- Eldh, Sigrid, 1961-, et al.
(författare)
-
Robustness Testing of Mobile Telecommunication Systems : A Case Study on Industrial Practice and Challenges
- 2012
-
Ingår i: Proceedings - IEEE 5th International Conference on Software Testing, Verification and Validation, ICST 2012. - : IEEE Press. - 9780769546704 ; , s. 895-900
-
Konferensbidrag (refereegranskat)abstract
- Robustness relates to the capability of a system to handle internal and external negative situations and disturbances. Robustness testing is the act of subjecting the system under test to such disturbances in a controlled manner. The objective of this study is to understand how robustness is considered in the development and testing of large-scale telecom systems, to identify the main challenges related to robustness and robustness testing, and to identify potential improvements to the current situation. We performed an exploratory case study of a telecom industry. Data was collected through interviews, study of documentation, and participant observation. Our result is a number of challenges related to robustness testing. The key challenge identified relates to understanding how robustness test can be broken down from a system-level to a low-level perspective. Our conclusion is that the area of robustness test is challenging for large complex systems, where the understanding of how to provoke complex failures and derive a root cause, as well as defining the correct level of robust design in software is not sufficiently explored.
|
|
| 16. |
|
|
| 17. |
- Enoiu, Eduard Paul, et al.
(författare)
-
Model-based test suite generation for function block diagrams using the UPPAAL model checker
- 2013
-
Ingår i: Proceedings - IEEE 6th International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2013. ; , s. 158-167
-
Konferensbidrag (refereegranskat)abstract
- A method for model-based test generation of safety-critical embedded applications using Programmable Logic Controllers and implemented in a programming language such as Function Block Diagram (FBD) is described. The FBD component model is based on the IEC 1131 standard and it is used primarily for embedded systems, in which timeliness is an important property to be tested. Our method involves the transformation of FBD programs with timed annotations into timed automata models which are used to automatically generate test suites. Specifically we demonstrate how to use model transformation for formalization and model checking of FBD programs using the UPPAAL tool. Many benefits emerge from this method, including the ability to automatically generate test suites from a formal model in order to ensure compliance to strict quality requirements including unit testing and specific coverage measurements. The approach is experimentally assessed on a train control system in terms of consumed resources.
|
|
| 18. |
|
|
| 19. |
- Enoiu, Eduard Paul, et al.
(författare)
-
MOS : An Integrated Model-based and Search-based Testing Tool for Function Block Diagrams
- 2013
-
Ingår i: 2013 1st International Workshop on Combining Modelling and Search-Based Software Engineering, CMSBSE 2013 - Proceedings. - 9781467362849 ; , s. 55-60
-
Konferensbidrag (refereegranskat)abstract
- In this paper we present a new testing tool for safety critical applications described in Function Block Diagram (FBD) language aimed to support both a model and a search-based approach. Many benefits emerge from this tool, including the ability to automatically generate test suites from an FBD program in order to comply to quality requirements such as component testing and specific coverage measurements. Search-based testing methods are used to generate test data based on executable code rather than the FBD program, alleviating any problems that may arise from the ambiguities that occur while creating FBD programs. Test cases generated by both approaches are executed and used as a way of cross validation. In the current work, we describe the architecture of the tool, its workflow process, and a case study in which the tool has been applied in a real industrial setting to test a train control management system.
|
|
| 20. |
- Enoiu, Eduard Paul, et al.
(författare)
-
Using Logic Coverage to Improve Testing Function Block Diagrams
- 2013
-
Ingår i: Testing Software and Systems. - Berlin, Heidelberg : Springer Berlin Heidelberg. - 9783642417061 ; , s. 1-16
-
Bokkapitel (refereegranskat)abstract
- In model-driven development, testers are often focusing on functional model-level testing, enabling verification of design models against their specifications. In addition, in safety-critical software development, testers are required to show that tests cover the structure of the implementation. Testing cost and time savings could be achieved if the process of deriving test cases for logic coverage is automated and provided test cases are ready to be executed. The logic coverage artifacts, i.e., predicates and clauses, are required for different logic coverage, e.g., MC/DC. One way of dealing with test case generation for ensuring logic coverage is to approach it as a model-checking problem, such that model-checking tools automatically create test cases. We show how logic coverage criteria can be formalized and used by a model-checker to provide test cases for ensuring this coverage on safety-critical software described in the Function Block Diagram programming language. Based on our experiments, this approach, supported by a tool chain, is an applicable and useful way of generating test cases for covering Function Block Diagrams.
|
|
| 21. |
|
|
| 22. |
- Kienle, Holger, et al.
(författare)
-
Liability for Software in Safety-Critical Mechatronic Systems : An Industrial Questionnaire
- 2012
-
Ingår i: 2012 2nd International Workshop on Software Engineering for Embedded Systems, SEES 2012 - Proceedings. - 9781467318532 ; , s. 44-50
-
Konferensbidrag (refereegranskat)abstract
- There is very little research on how industry is dealing with the risk of legal liability when constructing safety- critical mechatronic systems that are also software intensive. In this paper we propose a case study approach with the goal to understand how liability concerns in this setting impact software development in industry. The approach takes into account that software development is embedded into a complex socio-technical context involving stakeholders from technical, managerial and legal backgrounds. We present first results of our case study from a questionnaire involving six companies that develop software- intensive, safety-critical systems in the vehicular and avionics domains. The results of the questionnaire shed light on current industrial practices and concerns. The results indicate that liability seems indeed a concern and that a more in-depth analysis of this topic would be desirable to better understand the strategies that are used by industry to address liability risks.
|
|
| 23. |
|
|
| 24. |
- Pei-Breivold, Hongyu, et al.
(författare)
-
What Does Research Say About Agile and Architecture?
- 2010
-
Ingår i: Proceedings - 5th International Conference on Software Engineering Advances, ICSEA 2010. - 9780769541440 ; , s. 32-37
-
Konferensbidrag (refereegranskat)abstract
- Agile has been used to refer to a software development paradigm that emphasizes rapid and flexible development. In the meanwhile, we have through our practical experiences in scaling up agile methods, noticed that architecture plays an important role. Due to the inter-relationship between agile methods and architecture, as well as divergent perceptions on their correlation stated in numerous sources, we are motivated to find out how these perceptions are supported by findings in the research community in general and in empirical studies in particular. To fully benefit from agile practices and architectural disciplines, we need empirical data on the perceived and experienced impacts of introducing agile methods to existing software development process, as well as correlations between agile and architecture. In this paper, we survey the research literature for statements made regarding the relationship between agile development and software architecture. The main findings are that there is a lack of scientific support for many of the claims that are concerned with agile and architecture, and more empirical studies are needed to fully reveal the benefits and drawbacks implied by an agile software development method.
|
|
| 25. |
- Sundmark, Daniel, et al.
(författare)
-
An exploratory case study of testing in an automotive electrical system release process
- 2011
-
Ingår i: SIES 2011 - 6th IEEE International Symposium on Industrial Embedded Systems, Conference Proceedings. - Västerås : IEEE. - 9781612848204 ; , s. 166-175
-
Konferensbidrag (refereegranskat)abstract
- The release process is a crucial element in the development of software-intensive systems, as it bridges the gap between the development of a system and its operational use. A short release process enables a fast time to market, but also puts high demands on the efficiency of integration and testing, which typically constitue principal release process steps. This paper reports findings from an exploratory industrial case study focusing on system testing in an automotive electrical system release process. We provide a description of how system testing is performed and integrated in the release process in the automotive domain, and identify a set of challenges observed in the studied setting. The case being studied is Scania, a major Swedish automotive company. © 2011 IEEE.
|
|
| 26. |
- Wiklund, Kristian, et al.
(författare)
-
Can We Do Useful Industrial Software Engineering Research in the Shadow of Lean and Agile?
- 2013
-
Ingår i: 2013 1ST INTERNATIONAL WORKSHOP ON CONDUCTING EMPIRICAL STUDIES IN INDUSTRY (CESI). - : IEEE. ; , s. 67-
-
Konferensbidrag (refereegranskat)abstract
- The software industry is rapidly changing from traditional ways of working to lean and agile development methods using self-organized feature development teams that are performing a much larger part of the development process than before. Face to face communication will replace many of the design artifacts used for work-in-progress, such as defect reports and feature system design specifications. This type of data will cease to exist when the feature is developed or the problem is solved, and will not be readily available to researchers. As a consequence, software engineering research in industry will have to rely primarily on participatory and observational methods.
|
|
| 27. |
|
|
| 28. |
- Wiklund, Kristian, et al.
(författare)
-
Impediments for Automated Testing - An Empirical Analysis of a User Support Discussion Board
- 2014
-
Ingår i: Seventh IEEE International Conference on Software Testing, Verification and Validation ICST 2014. - : IEEE. - 9780769551852 ; , s. 113-122
-
Konferensbidrag (refereegranskat)abstract
- To better understand the challenges encountered by users and developers of automatic software testing, we have performed an empirical investigation of a discussion board used for support of a test automation framework having several hundred users. The messages on the discussion board were stratified into problem reports, help requests, development information, and feature requests. The messages in the problem report and help request strata were then sampled and analyzed using thematic analysis, searching for common patterns. Our analysis indicate that a large part of the impediments discussed on the board are related to issues related to the centralized IT environment, and to erroneous behavior connected to the use of the framework and related components. We also observed a large amount of impediments related to the use of software development tools. Turning to the help requests, we found that the majority of the help requests were about designing test scripts and not about the areas that appear to be most problematic. From our results and previous publications, we see a clear need to simplify the use, installation, and configuration of test systems of this type. The problems attributable to software development tools suggest that testers implementing test automation need more skills in handling those tools, than historically has been assumed. Finally, we propose that further research into the benefits of centralization of tools and IT environments, as well as structured deployment and efficient use of test automation, is performed.
|
|
| 29. |
- Wiklund, Kristian, et al.
(författare)
-
Impediments in Agile Software Development : An Empirical Investigation
- 2013
-
Ingår i: Product-Focused Software Process Improvement, Lecture Notes in Computer Science Volume 7983. - Berlin, Heidelberg : Springer. - 9783642392580 - 9783642392597 ; , s. 35-49
-
Bokkapitel (refereegranskat)abstract
- In this paper, we report on a case study on development impediments encountered in the early phase of a transformation to agile methods in a software development organization. Drawing from literature and anecdotal evidence, it was assumed that the majority of the impediments were related to software testing. To investigate this, we performed a case study seeking qualitative and quantitative evidence from task boards, interviews, and observations. Our analysis indicates that the major challenge in the transformation undertaken by the studied organization was coordination and communication in the large, and that testing was the major challenge only when the unit of analysis was restricted to the teams in the department.
|
|
| 30. |
|
|
| 31. |
- Wiklund, Kristian, 1972-, et al.
(författare)
-
Technical Debt in Test Automation
- 2012
-
Ingår i: Proceedings - IEEE 5th International Conference on Software Testing, Verification and Validation, ICST 2012. - : IEEE Computer Society. - 9780769546704 ; , s. 887-892
-
Konferensbidrag (refereegranskat)abstract
- Automated test execution is one of the more popular and available strategies to minimize the cost for software testing, and is also becoming one of the central concepts in modern software development as methods such as test-driven devel- opment gain popularity. Published studies on test automation indicate that the maintenance and development of test automation tools commonly encounter problems due to unforeseen issues. To further investigate this, we performed a case study on a telecommunication subsystem to seek factors that contribute to inefficiencies in use, maintenance, and development of the automated testing performed within the scope of responsibility of a software design team. A qualitative evaluation of the findings indicates that the main areas of improvement in this case are in the fields of interaction design and general software design principles, as applied to test execution system development.
|
|
