| 1. |
|
|
| 2. |
- Alégroth, Emil, 1984, et al.
(författare)
-
Conceptualization and evaluation of component-based testing unified with visual GUI testing: An empirical study
- 2015
-
Ingår i: 2015 IEEE 8th International Conference on Software Testing, Verification and Validation, ICST 2015 - Proceedings. - 2159-4848. - 9781479971251
-
Konferensbidrag (refereegranskat)abstract
- In this paper we present the results of a two-phase empirical study where we evaluate and compare the applicability of automated component-based Graphical User Interface (GUI) testing and Visual GUI Testing (VGT) in the tools GUITAR and a prototype tool we refer to as VGT GUITAR. First, GUI mutation operators are defined to create 18 faulty versions of an application on which both tools are then applied in an experiment. Results from 456 test case executions in each tool show, with statistical significance, that the component-based approach reports more false negatives than VGT for acceptance tests but that the VGT approach reports more false positives for system tests. Second, a case study is performed with larger open source applications, ranging from 8,803-55,006 lines of code. Results show that GUITAR is applicable in practice but has some challenges related to GUI component states. The results also show that VGT GUITAR is currently not applicable in practice and therefore requires further research and development. Based on the study's results we present areas of future work for both test approaches and conclude that the approaches have different benefits and drawbacks. The component-based approach is robust and executes tests faster than the VGT approach, with a factor of 3. However, the VGT approach can perform visual assertions and is perceived more flexible than the component- based approach. These conclusions let us hypothesize that a combination of the two approaches is the most suitable in practice and therefore warrants future research.
|
|
| 3. |
- Alégroth, Emil, et al.
(författare)
-
Continuous Integration and Visual GUI Testing : Benefits and Drawbacks in Industrial Practice
- 2018
-
Ingår i: Proceedings - 2018 IEEE 11th International Conference on Software Testing, Verification and Validation, ICST 2018. - : Institute of Electrical and Electronics Engineers Inc.. - 9781538650127 ; , s. 172-181
-
Konferensbidrag (refereegranskat)abstract
- Continuous integration (CI) is growing in industrial popularity, spurred on by market trends towards faster delivery and higher quality software. A key facilitator of CI is automated testing that should be executed, automatically, on several levels of system abstraction. However, many systems lack the interfaces required for automated testing. Others lack test automation coverage of the system under test's (SUT) graphical user interface (GUI) as it is shown to the user. One technique that shows promise to solve these challenges is Visual GUI Testing (VGT), which uses image recognition to stimulate and assert the SUT's behavior. Research has presented the technique's applicability and feasibility in industry but only limited support, from an academic setting, that the technique is applicable in a CI environment. In this paper we presents a study from an industrial design research study with the objective to help bridge the gap in knowledge regarding VGT's applicability in a CI environment in industry. Results, acquired from interviews, observations and quantitative analysis of 17.567 test executions, collected over 16 weeks, show that VGT provides similar benefits to other automated test techniques for CI. However, several significant drawbacks, such as high costs, are also identified. The study concludes that, although VGT is applicable in an industrial CI environment, its severe challenges require more research and development before the technique becomes efficient in practice. © 2018 IEEE.
|
|
| 4. |
- Alégroth, Emil, 1984, et al.
(författare)
-
Exploring the Presence of Technical Debt in Industrial GUI-based Testware: A Case Study
- 2016
-
Ingår i: 2016 IEEE Ninth International Conference on Software Testing, Verification and Validation Workshops. - : IEEE. - 2159-4848. - 9781509036745 ; , s. 257-262
-
Konferensbidrag (refereegranskat)abstract
- Technical debt (TD) is a concept used to describe a sub-optimal solution of a software artifact that negatively affects its comprehensibility, extendability and maintainability. As such, TD adversely affects the costs or quality associated with the artifact, which is also called interest. TD has through research been identified in all types of software artifacts, from architectural design to automated tests (Testware). However, research into testware technical debt (TTD) is limited and primarily focused on testing on lower level of system abstraction, i.e. unit-and integration tests, leaving a need for more TTD research on GUI-based testing. In this study we explore this gap in knowledge through an industrial case study at a Swedish avionics software development company. Four repositories are studied for the presence of TTD using expert interviews, semi-automated document analysis and automatic metric analysis. Results of the study provide initial support that the concept of TTD is applicable to GUI-based testware and show the presence of both TD items unique to GUI-based testware and items common to software. The implications of these results are that engineering best practices must be established for GUI-based testware to minimize TD interest.
|
|
| 5. |
- Alégroth, Emil, 1984, et al.
(författare)
-
Maintenance of automated test suites in industry: An empirical study on Visual GUI Testing
- 2016
-
Ingår i: Information and Software Technology. - : Elsevier BV. - 0950-5849 .- 1873-6025. ; 73, s. 66-80
-
Tidskriftsartikel (refereegranskat)abstract
- Context: Verification and validation (V&V) activities make up 20-50% of the total development costs of a software system in practice. Test automation is proposed to lower these V&V costs but available research only provides limited empirical data from industrial practice about the maintenance costs of automated tests and what factors affect these costs. In particular, these costs and factors are unknown for automated GUI-based testing. Objective: This paper addresses this lack of knowledge through analysis of the costs and factors associated with the maintenance of automated GUI-based tests in industrial practice. Method: An empirical study at two companies, Siemens and Saab, is reported where interviews about, and empirical work with, Visual GUI Testing is performed to acquire data about the technique's maintenance costs and feasibility. Results: 13 factors are observed that affect maintenance, e.g. tester knowledge/experience and test case complexity. Further, statistical analysis shows that developing new test scripts is costlier than maintenance but also that frequent maintenance is less costly than infrequent, big bang maintenance. In addition a cost model, based on previous work, is presented that estimates the time to positive return on investment (ROI) of test automation compared to manual testing. Conclusions: It is concluded that test automation can lower overall software development costs of a project while also having positive effects on software quality. However, maintenance costs can still be considerable and the less time a company currently spends on manual testing, the more time is required before positive, economic, ROI is reached after automation.
|
|
| 6. |
- Alégroth, Emil, 1984, et al.
(författare)
-
On the industrial applicability of TextTest: An empirical case study
- 2015
-
Ingår i: 2015 IEEE 8th International Conference on Software Testing, Verification and Validation, ICST 2015 - Proceedings. - 2159-4848. - 9781479971251
-
Konferensbidrag (refereegranskat)abstract
- Software systems are becoming more complex, not least in their Graphical User Interfaces (GUIs), which presents challenges for existing testing practices. Pressure to reduce time to market leaves less time for manual testing and increases the importance of test automation. Previous research has identified several generations of automated GUI-based test approaches with different cost-benefit tradeoffs. Whilst test automation provides fast quality feedback it can be associated with high costs and inability to identify defects not explicitly anticipated by the test designer. TextTest is a capture-replay tool for GUI-based testing with a novel approach that overcomes several of the challenges experienced with previous approaches. Firstly the tool supports Approval Testing, an approach where ASCII-art representations of the GUI's visual state are used to verify correct application behavior at the system level. Secondly it records and replays test scripts in a user defined domain specific language (DSL) that is readable by all stakeholders. In this paper we present a three phase industrial case study that aims to identify TextTest's applicability in industrial practice. The paper reports that the tool is associated with (1) low script development costs due to recording functionality, (2) low maintenance costs, on average 7 minutes per test case, (3) better defect finding ability than manual system testing, (4) high test case execution performance (In this case 500 test cases in 20 minutes), (5) high script readability due to DSL defined scripts, and (6) test suites that are robust to change (In this case 93 percent per iteration). However, the tool requires a higher degree of technical skill for customization work, test maintainers need skills in designing regular expressions and the tool's applicability is currently restricted to Java and Python based applications.
|
|
| 7. |
- Alégroth, Emil, et al.
(författare)
-
On the long-term use of visual gui testing in industrial practice : a case study
- 2017
-
Ingår i: Empirical Software Engineering. - : Springer. - 1382-3256 .- 1573-7616. ; 22:6, s. 2937-2971
-
Tidskriftsartikel (refereegranskat)abstract
- Visual GUI Testing (VGT) is a tool-driven technique for automated GUI-based testing that uses image recognition to interact with and assert the correctness of the behavior of a system through its GUI as it is shown to the user. The technique’s applicability, e.g. defect-finding ability, and feasibility, e.g. time to positive return on investment, have been shown through empirical studies in industrial practice. However, there is a lack of studies that evaluate the usefulness and challenges associated with VGT when used long-term (years) in industrial practice. This paper evaluates how VGT was adopted, applied and why it was abandoned at the music streaming application development company, Spotify, after several years of use. A qualitative study with two workshops and five well chosen employees is performed at the company, supported by a survey, which is analyzed with a grounded theory approach to answer the study’s three research questions. The interviews provide insights into the challenges, problems and limitations, but also benefits, that Spotify experienced during the adoption and use of VGT. However, due to the technique’s drawbacks, VGT has been abandoned for a new technique/framework, simply called the Test interface. The Test interface is considered more robust and flexible for Spotify’s needs but has several drawbacks, including that it does not test the actual GUI as shown to the user like VGT does. From the study’s results it is concluded that VGT can be used long-term in industrial practice but it requires organizational change as well as engineering best practices to be beneficial. Through synthesis of the study’s results, and results from previous work, a set of guidelines are presented that aim to aid practitioners to adopt and use VGT in industrial practice. However, due to the abandonment of the technique, future research is required to analyze in what types of projects the technique is, and is not, long-term viable. To this end, we also present Spotify’s Test interface solution for automated GUI-based testing and conclude that it has its own benefits and drawbacks.
|
|
| 8. |
- Alégroth, Emil, et al.
(författare)
-
Overview of the ICST International Software Testing Contest
- 2017
-
Ingår i: Proceedings - 10th IEEE International Conference on Software Testing, Verification and Validation, ICST 2017. - : IEEE Computer Society. - 9781509060313 ; , s. 550-551
-
Konferensbidrag (refereegranskat)abstract
- In the software testing contest, practitioners and researcher's are invited to test their test approaches against similar approaches to evaluate pros and cons and which is perceivably the best. The 2017 iteration of the contest focused on Graphical User Interface-driven testing, which was evaluated on the testing tool TESTONA. The winner of the competition was announced at the closing ceremony of the international conference on software testing (ICST), 2017. © 2017 IEEE.
|
|
| 9. |
- Alégroth, Emil, et al.
(författare)
-
Replicating Rare Software Failures with Exploratory Visual GUI Testing
- 2017
-
Ingår i: IEEE Software. - : IEEE Computer Society. - 0740-7459 .- 1937-4194. ; 34:5, s. 53-59
-
Tidskriftsartikel (refereegranskat)abstract
- Saab AB developed software that had a defect that manifested itself only after months of continuous system use. After years of customer failure reports, the defect still persisted, until Saab developed failure replication based on visual GUI testing. © 1984-2012 IEEE.
|
|
| 10. |
- Alégroth, Emil, 1984, et al.
(författare)
-
Teaching scrum – what we did, what we will do and what impedes us
- 2015
-
Ingår i: Lecture Notes in Business Information Processing. - 1865-1348 .- 1865-1356. - 9783319186115 ; 212, s. 361-362
-
Konferensbidrag (övrigt vetenskapligt/konstnärligt)abstract
- This paper analyses the way we teach Scrum. We reflect on our intended learning outcomes, which challenges we find in teaching Scrum and which lessons we have learned during the last four years. We also give an outlook on the way we want to introduce and apply Scrum in our teaching and how we intend to improve the curriculum.
|
|
