| 1. |
- Alégroth, Emil, 1984, et al.
(författare)
-
Conceptualization and evaluation of component-based testing unified with visual GUI testing: An empirical study
- 2015
-
Ingår i: 2015 IEEE 8th International Conference on Software Testing, Verification and Validation, ICST 2015 - Proceedings. - 2159-4848. - 9781479971251
-
Konferensbidrag (refereegranskat)abstract
- In this paper we present the results of a two-phase empirical study where we evaluate and compare the applicability of automated component-based Graphical User Interface (GUI) testing and Visual GUI Testing (VGT) in the tools GUITAR and a prototype tool we refer to as VGT GUITAR. First, GUI mutation operators are defined to create 18 faulty versions of an application on which both tools are then applied in an experiment. Results from 456 test case executions in each tool show, with statistical significance, that the component-based approach reports more false negatives than VGT for acceptance tests but that the VGT approach reports more false positives for system tests. Second, a case study is performed with larger open source applications, ranging from 8,803-55,006 lines of code. Results show that GUITAR is applicable in practice but has some challenges related to GUI component states. The results also show that VGT GUITAR is currently not applicable in practice and therefore requires further research and development. Based on the study's results we present areas of future work for both test approaches and conclude that the approaches have different benefits and drawbacks. The component-based approach is robust and executes tests faster than the VGT approach, with a factor of 3. However, the VGT approach can perform visual assertions and is perceived more flexible than the component- based approach. These conclusions let us hypothesize that a combination of the two approaches is the most suitable in practice and therefore warrants future research.
|
|
| 2. |
- Alégroth, Emil, 1984, et al.
(författare)
-
On the industrial applicability of TextTest: An empirical case study
- 2015
-
Ingår i: 2015 IEEE 8th International Conference on Software Testing, Verification and Validation, ICST 2015 - Proceedings. - 2159-4848. - 9781479971251
-
Konferensbidrag (refereegranskat)abstract
- Software systems are becoming more complex, not least in their Graphical User Interfaces (GUIs), which presents challenges for existing testing practices. Pressure to reduce time to market leaves less time for manual testing and increases the importance of test automation. Previous research has identified several generations of automated GUI-based test approaches with different cost-benefit tradeoffs. Whilst test automation provides fast quality feedback it can be associated with high costs and inability to identify defects not explicitly anticipated by the test designer. TextTest is a capture-replay tool for GUI-based testing with a novel approach that overcomes several of the challenges experienced with previous approaches. Firstly the tool supports Approval Testing, an approach where ASCII-art representations of the GUI's visual state are used to verify correct application behavior at the system level. Secondly it records and replays test scripts in a user defined domain specific language (DSL) that is readable by all stakeholders. In this paper we present a three phase industrial case study that aims to identify TextTest's applicability in industrial practice. The paper reports that the tool is associated with (1) low script development costs due to recording functionality, (2) low maintenance costs, on average 7 minutes per test case, (3) better defect finding ability than manual system testing, (4) high test case execution performance (In this case 500 test cases in 20 minutes), (5) high script readability due to DSL defined scripts, and (6) test suites that are robust to change (In this case 93 percent per iteration). However, the tool requires a higher degree of technical skill for customization work, test maintainers need skills in designing regular expressions and the tool's applicability is currently restricted to Java and Python based applications.
|
|
| 3. |
- Erman, Nicklas, et al.
(författare)
-
Navigating Information Overload Caused by Automated Testing – A Clustering Approach in Multi-Branch Development
- 2015
-
Ingår i: 2015 IEEE 8th International Conference on Software Testing, Verification and Validation, ICST 2015 - Proceedings. - 9781479971251
-
Konferensbidrag (refereegranskat)abstract
- Background. Test automation is a widely used technique to increase the efficiency of software testing. However, executing more test cases increases the effort required to analyze test results. At Qlik, automated tests run nightly for up to 20 development branches, each containing thousands of test cases, resulting in information overload. Aim. We therefore develop a tool that supports the analysis of test results. Method. We create NIOCAT, a tool that clusters similar test case failures, to help the analyst identify underlying causes. To evaluate the tool, experiments on manually created subsets of failed test cases representing different use cases are conducted, and a focus group meeting is held with test analysts at Qlik. Results. The case study shows that NIOCAT creates accurate clusters, in line with analyses performed by human analysts. Further, the potential time-savings of our approach is confirmed by the participants in the focus group. Conclusions. NIOCAT provides a feasible complement to current automated testing practices at Qlik by reducing information overload.
|
|
| 4. |
- Poulding, Simon, et al.
(författare)
-
Re-using generators of complex test data
- 2015
-
Ingår i: 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST). - : IEEE Computer Society. - 9781479971251 ; , s. Article number 7102605-
-
Konferensbidrag (refereegranskat)abstract
- The efficiency of random testing can be improved by sampling test inputs using a generating program that incorporates knowledge about the types of input most likely to detect faults in the software-under-test (SUT). But when the input of the SUT is a complex data type - such as a domain-specific string, array, record, tree, or graph - creating such a generator may be time- consuming and may require the tester to have substantial prior experience of the domain. In this paper we propose the re-use of generators created for one SUT on other SUTs that take the same complex data type as input. The re-use of a generator in this way would have little overhead, and we hypothesise that the re-used generator will typically be as least as efficient as the most straightforward form of random testing: sampling test inputs from the uniform distribution. We investigate this proposal for two data types using five generators. We assess test efficiency against seven real-world SUTs, and in terms of both structural coverage and the detection of seeded faults. The results support the re-use of generators for complex data types, and suggest that if a library of generators is to be maintained for this purpose, it is possible to extend library generators to accommodate the specific testing requirements of newly-encountered SUTs. © 2015 IEEE.
|
|
| 5. |
- Yamada, Akihisa, et al.
(författare)
-
Optimization of Combinatorial Testing by Incremental SAT Solving
- 2015
-
Ingår i: 2015 IEEE 8th International Conference on Software Testing, Verification and Validation, ICST 2015 - Proceedings. - : Institute of Electrical and Electronics Engineers (IEEE). - 9781479971251 ; , s. 1-10
-
Konferensbidrag (refereegranskat)abstract
- Combinatorial testing aims at reducing the cost of software and system testing by reducing the number of test cases to be executed. We propose an approach for combinatorial testing that generates a set of test cases that is as small as possible, using incremental SAT solving. We present several search-space pruning techniques that further improve our approach. Experiments show a significant improvement of our approach over other SAT-based approaches, and considerable reduction of the number of test cases over other combinatorial testing tools.
|
|
