Maintenance of Automated Test Suites in Industry: An Empirical study on Visual GUI Testing

Context: Verification and validation (V&V) activities make up 20 to 50 percent of the total development costs of a software system in practice. Test automation is proposed to lower these V&V costs but available research only provides limited empirical data from industrial practice about the maintenance costs of automated tests and what factors affect these costs. In particular, these costs and factors are unknown for automated GUI-based testing. Objective: This paper addresses this lack of knowledge through analysis of the costs and factors associated with the maintenance of automated GUI-based tests in industrial practice. Method: An empirical study at two companies, Siemens and Saab, is reported where interviews about, and empirical work with, Visual GUI Testing is performed to acquire data about the technique's maintenance costs and feasibility. Results: 13 factors are observed that affect maintenance, e.g. tester knowledge/experience and test case complexity. Further, statistical analysis shows that developing new test scripts is costlier than maintenance but also that frequent maintenance is less costly than infrequent, big bang maintenance. In addition a cost model, based on previous work, is presented that estimates the time to positive return on investment (ROI) of test automation compared to manual testing. Conclusions: It is concluded that test automation can lower overall software development costs of a project whilst also having positive effects on software quality. However, maintenance costs can still be considerable and the less time a company currently spends on manual testing, the more time is required before positive, economic, ROI is reached after automation

Trade-off between automated and manual testing: A production possibility curve cost model

Testing is always important for Software Quality Assurance (SQA) activities and key cost multiplier in software development. The decision to automate or not to automate a test case is critical. In this paper we discuss the possibility of test automation and in relation to the trade-off between manual and automated test cases. We purpose a Production cost frontier based technique to distinguish the point of automation and manual test within the cost constraints. Our objective is to identify the facts that up to what extant a testing process can be automated. In this paper a cost model is proposed for deciding the proportion of automated and manual testing. The objective is to find best possible combination of these two and production possibility in one type by eliminating the other type of testin

Evolution and Fragility of Mobile Automated Test Suites

L'abstract è presente nell'allegato / the abstract is in the attachmen