| 1. |
- Pernstal, J., et al.
(författare)
-
FLEX-RCA: a lean-based method for root cause analysis in software process improvement
- 2019
-
Ingår i: Software Quality Journal. - : Springer Science and Business Media LLC. - 1573-1367 .- 0963-9314. ; 27:1, s. 389-428
-
Tidskriftsartikel (refereegranskat)abstract
- Software process improvement (SPI) is an instrument to increase the productivity of, and the quality of work, in software organizations. However, a majority of SPI frameworks are too extensive or provide guidance and potential improvement areas at a high level, indicating only the symptoms, not the causes. Motivated by the industrial need of two Swedish automotive companies to systematically uncover the underlying root causes of high-level improvement issues identified in an SPI project-assessing inter-departmental interactions in large-scale software systems development-this paper advances a root cause analysis (RCA) method building on Lean Six Sigma, called Flex-RCA. Flex-RCA is used to delve deeper into challenges identified to find root causes as a part of the evaluation and subsequent improvement activities. We also demonstrate and evaluate Flex-RCA's industrial applicability in a case study. An overall conclusion is that the use of Flex-RCA was successful, showing that it had the desired effect of both producing a broad base of causes on a high level and, more importantly, enabling an exploration of the underlying root causes.
|
|
| 2. |
- Pernstål, Joakim, et al.
(författare)
-
Communication Problems in Software Development : A Model and Its Industrial Application
- 2019
-
Ingår i: International journal of software engineering and knowledge engineering. - : World Scientific Publishing Co. Pte Ltd. - 0218-1940. ; 29:10, s. 1497-1538
-
Tidskriftsartikel (refereegranskat)abstract
- Attaining effective communication within and across organizational units is among the most critical challenges for success in software development organizations. This paper presents a novel model, supporting analysis of problems in inter-departmental communication events. The model was developed and designed based on industrial needs emphasizing flexibility, applicability and scalability. The model covers central communication aspects in order to provide a useful approximation of communication problems rather than in-depth modeling on message-by message basis. Other event-specific information, such as costs, can then be attached to enrich analysis and understanding. To exemplify and evaluate the model and collect feedback from industry, it was applied to 16 events at a Swedish automotive manufacturer where communication between two departments had broken down during development of software-intensive systems. The evaluation showed that the model helped structure and conduct systematic data collection and analysis of dysfunctional communication patterns. We found that insufficient understanding of the matters being communicated was prevalent, but also more specifically, requirements were insufficiently balanced, detailed and specified over the full system development cycle. Besides, the long-term cost for the company was analyzed in depth for each event, yielding a total estimated cost for the analyzed communication events of 11.2MUS$. © 2019 World Scientific Publishing Company.
|
|
| 3. |
- Alégroth, Emil, 1984, et al.
(författare)
-
Maintenance of automated test suites in industry: An empirical study on Visual GUI Testing
- 2016
-
Ingår i: Information and Software Technology. - : Elsevier BV. - 0950-5849 .- 1873-6025. ; 73, s. 66-80
-
Tidskriftsartikel (refereegranskat)abstract
- Context: Verification and validation (V&V) activities make up 20-50% of the total development costs of a software system in practice. Test automation is proposed to lower these V&V costs but available research only provides limited empirical data from industrial practice about the maintenance costs of automated tests and what factors affect these costs. In particular, these costs and factors are unknown for automated GUI-based testing. Objective: This paper addresses this lack of knowledge through analysis of the costs and factors associated with the maintenance of automated GUI-based tests in industrial practice. Method: An empirical study at two companies, Siemens and Saab, is reported where interviews about, and empirical work with, Visual GUI Testing is performed to acquire data about the technique's maintenance costs and feasibility. Results: 13 factors are observed that affect maintenance, e.g. tester knowledge/experience and test case complexity. Further, statistical analysis shows that developing new test scripts is costlier than maintenance but also that frequent maintenance is less costly than infrequent, big bang maintenance. In addition a cost model, based on previous work, is presented that estimates the time to positive return on investment (ROI) of test automation compared to manual testing. Conclusions: It is concluded that test automation can lower overall software development costs of a project while also having positive effects on software quality. However, maintenance costs can still be considerable and the less time a company currently spends on manual testing, the more time is required before positive, economic, ROI is reached after automation.
|
|
| 4. |
- Alégroth, Emil, 1984, et al.
(författare)
-
Visual GUI testing in practice: challenges, problemsand limitations
- 2015
-
Ingår i: Empirical Software Engineering. - : Springer Science and Business Media LLC. - 1573-7616 .- 1382-3256. ; 20:3, s. 694-744
-
Tidskriftsartikel (refereegranskat)abstract
- In today's software development industry, high-level tests such as Graphical User Interface (GUI) based system and acceptance tests are mostly performed with manual practices that are often costly, tedious and error prone. Test automation has been proposed to solve these problems but most automation techniques approach testing from a lower level of system abstraction. Their suitability for high-level tests has therefore been questioned. High-level test automation techniques such as Record and Replay exist, but studies suggest that these techniques suffer from limitations, e.g. sensitivity to GUI layout or code changes, system implementation dependencies, etc. Visual GUI Testing (VGT) is an emerging technique in industrial practice with perceived higher flexibility and robustness to certain GUI changes than previous high-level (GUI) test automation techniques. The core of VGT is image recognition which is applied to analyze and interact with the bitmap layer of a system's front end. By coupling image recognition with test scripts, VGT tools can emulate end user behavior on almost any GUI-based system, regardless of implementation language, operating system or platform. However, VGT is not without its own challenges, problems and limitations (CPLs) but, like for many other automated test techniques, there is a lack of empirically-based knowledge of these CPLs and how they impact industrial applicability. Crucially, there is also a lack of information on the cost of applying this type of test automation in industry. This manuscript reports an empirical, multi-unit case study performed at two Swedish companies that develop safety-critical software. It studies their transition from manual system test cases into tests automated with VGT. In total, four different test suites that together include more than 300 high-level system test cases were automated for two multi-million lines of code systems. The results show that the transitioned test cases could find defects in the tested systems and that all applicable test cases could be automated. However, during these transition projects a number of hurdles had to be addressed; a total of 58 different CPLs were identified and then categorized into 26 types. We present these CPL types and an analysis of the implications for the transition to and use of VGT in industrial software development practice. In addition, four high-level solutions are presented that were identified during the study, which would address about half of the identified CPLs. Furthermore, collected metrics on cost and return on investment of the VGT transition are reported together with information about the VGT suites' defect finding ability. Nine of the identified defects are reported, 5 of which were unknown to testers with extensive experience from using the manual test suites. The main conclusion from this study is that even though there are many challenges related to the transition and usage of VGT, the technique is still valuable, flexible and considered cost-effective by the industrial practitioners. The presented CPLs also provide decision support in the use and advancement of VGT and potentially other automated testing techniques similar to VGT, e.g. Record and Replay.
|
|
| 5. |
- Berntsson Svensson, Richard, 1978, et al.
(författare)
-
The unfulfilled potential of data-driven decision making in agile software development
- 2019
-
Ingår i: Lecture Notes in Business Information Processing. - Cham : Springer International Publishing. - 1865-1356 .- 1865-1348. - 9783030190330 ; 355, s. 69-85
-
Konferensbidrag (refereegranskat)abstract
- With the general trend towards data-driven decision making (DDDM), organizations are looking for ways to use DDDM to improve their decisions. However, few studies have looked into the practitioners view of DDDM, in particular for agile organizations. In this paper we investigated the experiences of using DDDM, and how data can improve decision making. An emailed questionnaire was sent out to 124 industry practitioners in agile software developing companies, of which 84 answered. The results show that few practitioners indicated a wide-spread use of DDDM in their current decision making practices. The practitioners were more positive to its future use for higher-level and more general decision making, fairly positive to its use for requirements elicitation and prioritization decisions, while being less positive to its future use at the team level. The practitioners do see a lot of potential for DDDM in an agile context; however, currently unfulfilled.
|
|
| 6. |
- Dobslaw, Felix, 1983, et al.
(författare)
-
Estimating Return on Investment for GUI Test Automation Frameworks
- 2019
-
Ingår i: Proceedings - International Symposium on Software Reliability Engineering, ISSRE. - 1071-9458. - 9781728149813 ; 2019-October
-
Konferensbidrag (refereegranskat)abstract
- Automated graphical user interface (GUI) tests can reduce manual testing activities and increase test frequency. This motivates the conversion of manual test cases into automated GUI tests. However, it is not clear whether such automation is cost-effective given that GUI automation scripts add to the code base and demand maintenance as a system evolves. In this paper, we introduce a method for estimating maintenance cost and Return on Investment (ROI) for Automated GUI Testing (AGT). The method utilizes the existing source code change history and has the potential to be used for the evaluation of other testing or quality assurance automation technologies. We evaluate the method for a real-world, industrial software system and compare two fundamentally different AGT frameworks, namely Selenium and EyeAutomate, to estimate and compare their ROI. We also report on their defect-finding capabilities and usability. The quantitative data is complemented by interviews with employees at the company the study has been conducted at. The method was successfully applied, and estimated maintenance cost and ROI for both frameworks are reported. Overall, the study supports earlier results showing that implementation time is the leading cost for introducing AGT. The findings further suggest that, while EyeAutomate tests are significantly faster to implement, Selenium tests require more of a programming background but less maintenance.
|
|
| 7. |
|
|
| 8. |
|
|
| 9. |
- Feldt, Robert, 1972, et al.
(författare)
-
Message from the ICST 2018 Program Chairs
- 2018
-
Ingår i: Proceedings - 2018 IEEE 11th International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2018.
-
Konferensbidrag (övrigt vetenskapligt/konstnärligt)
|
|
| 10. |
- Feldt, Robert, 1972, et al.
(författare)
-
Searching for test data with feature diversity
- 2017
-
Tidskriftsartikel (övrigt vetenskapligt/konstnärligt)abstract
- There is an implicit assumption in software testing that more diverse and varied test data is needed for effective testing and to achieve different types and levels of coverage. Generic approaches based on information theory to measure and thus, implicitly, to create diverse data have also been proposed. However, if the tester is able to identify features of the test data that are important for the particular domain or context in which the testing is being performed, the use of generic diversity measures such as this may not be sufficient nor efficient for creating test inputs that show diversity in terms of these features. Here we investigate different approaches to find data that are diverse according to a specific set of features, such as length, depth of recursion etc. Even though these features will be less general than measures based on information theory, their use may provide a tester with more direct control over the type of diversity that is present in the test data. Our experiments are carried out in the context of a general test data generation framework that can generate both numerical and highly structured data. We compare random sampling for feature-diversity to different approaches based on search and find a hill climbing search to be efficient. The experiments highlight many trade-offs that needs to be taken into account when searching for diversity. We argue that recurrent test data generation motivates building statistical models that can then help to more quickly achieve feature diversity.
|
|
