Maintenance of Automated Test Suites in Industry: An Empirical study on Visual GUI Testing

Context: Verification and validation (V&V) activities make up 20 to 50 percent of the total development costs of a software system in practice. Test automation is proposed to lower these V&V costs but available research only provides limited empirical data from industrial practice about the maintenance costs of automated tests and what factors affect these costs. In particular, these costs and factors are unknown for automated GUI-based testing. Objective: This paper addresses this lack of knowledge through analysis of the costs and factors associated with the maintenance of automated GUI-based tests in industrial practice. Method: An empirical study at two companies, Siemens and Saab, is reported where interviews about, and empirical work with, Visual GUI Testing is performed to acquire data about the technique's maintenance costs and feasibility. Results: 13 factors are observed that affect maintenance, e.g. tester knowledge/experience and test case complexity. Further, statistical analysis shows that developing new test scripts is costlier than maintenance but also that frequent maintenance is less costly than infrequent, big bang maintenance. In addition a cost model, based on previous work, is presented that estimates the time to positive return on investment (ROI) of test automation compared to manual testing. Conclusions: It is concluded that test automation can lower overall software development costs of a project whilst also having positive effects on software quality. However, maintenance costs can still be considerable and the less time a company currently spends on manual testing, the more time is required before positive, economic, ROI is reached after automation

HTML5-pohjaisten hybridimobiilisovellusten käyttöliittymätestauksen automatisoinnin hyödyllisyys: Tapaustutkimus

While the first research papers on GUI test automation date back to the 1990s, its use in the software industry is still relatively rare. Traditionally, GUIs have been tested manually only and software automation has focused on the lower levels of testing such as unit testing or integration testing. The main reason for that is the complexity of GUIs compared to the lower software components. However, the manual testing of GUIs is a tedious and time-consuming process that involves repetitive and dull tasks, since the same tests need to be executed repeatedly on every testing iteration of the software under testing. The main goal with GUI test automation is to automate those steps and by doing so improve the cost-efficiency of the testing process and free the testers’ time on more meaningful and useful tasks. The previous research on GUI test automation reveals contradicting results. Some of the research has found GUI test automation to be both beneficial and cost-efficient and while others have found results suggesting the exact opposite. The contradicting results from previous research and the unclarity on the benefits, challenges, limitations and impediments of GUI test automation worked as the main driver for this thesis. The research was conducted as a combination of a literature review on the subject and a case study of three HTML5-based hybrid mobile application projects in the mobile development unit of one of the biggest IT companies in Finland.Käyttöliittymätestiautomaation käyttö yrityksissä on verrattain harvinaista, vaikka ensimmäiset tutkimukset aiheesta ovat 1990-luvulta. Perinteisesti käyttöliittymiä on testattu manuaalisesti ja ohjelmistotestausautomaatio keskitetty ohjelmiston alempien tasojen testaamiseen yksikkö- ja integraatiotesteillä. Pääsyy tähän on käyttöliittymien monimutkaisuus verrattuna alemman tason ohjelmistokomponentteihin. Käyttöliittymän testaaminen manuaalisesti on kuitenkin vaivalloinen, aikaa vaativa ja toisteinen prosessi, koska samat testit suoritetaan jokaisella testiajolla. Käyttöliittymätestauksen automatisoinnin päätavoite on testauksen kustannus-tehokkuuden parantaminen ja testaajien ajan vapauttaminen olennaisempiin tehtäviin. Aikaisemmat tutkimustulokset käyttöliittymätestiautomaatioon liittyen ovat ristiriitaisia. Osa tutkimuksista on todennut käyttöliittymätestiautomaation olevan hyödyllistä ja kustannustehokasta ja osa on päätynyt päinvastaisiin tuloksiin. Tämän työn päämotivaattoreina toimivat aiemman tutkimuksen ristiriitaiset tulokset ja epäselvyys käyttöliittymätestiautomaation hyödyllisyydestä ja kustannustehokkuudesta. Työn päätavoitteena oli tutkia voiko käyttöliittymätestiautomaation käyttö olla hyödyllistä ja kustannustehokasta. Työ koostuu kirjallisuuskatsauksesta ja kolmen HTML5-pohjaisen hybridimobiilisovelluksen tapaustutkimuksesta testiautomaation hyödyllisyyteen ja kustannustehokkuuteen liittyen

Translating Video Recordings of Mobile App Usages into Replayable Scenarios

Screen recordings of mobile applications are easy to obtain and capture a wealth of information pertinent to software developers (e.g., bugs or feature requests), making them a popular mechanism for crowdsourced app feedback. Thus, these videos are becoming a common artifact that developers must manage. In light of unique mobile development constraints, including swift release cycles and rapidly evolving platforms, automated techniques for analyzing all types of rich software artifacts provide benefit to mobile developers. Unfortunately, automatically analyzing screen recordings presents serious challenges, due to their graphical nature, compared to other types of (textual) artifacts. To address these challenges, this paper introduces V2S, a lightweight, automated approach for translating video recordings of Android app usages into replayable scenarios. V2S is based primarily on computer vision techniques and adapts recent solutions for object detection and image classification to detect and classify user actions captured in a video, and convert these into a replayable test scenario. We performed an extensive evaluation of V2S involving 175 videos depicting 3,534 GUI-based actions collected from users exercising features and reproducing bugs from over 80 popular Android apps. Our results illustrate that V2S can accurately replay scenarios from screen recordings, and is capable of reproducing $\approx$ 89% of our collected videos with minimal overhead. A case study with three industrial partners illustrates the potential usefulness of V2S from the viewpoint of developers.Comment: In proceedings of the 42nd International Conference on Software Engineering (ICSE'20), 13 page

Model-based testing in practice: An experience report from the web applications domain

publishedVersio

How do Developers Test Android Applications?

Enabling fully automated testing of mobile applications has recently become an important topic of study for both researchers and practitioners. A plethora of tools and approaches have been proposed to aid mobile developers both by augmenting manual testing practices and by automating various parts of the testing process. However, current approaches for automated testing fall short in convincing developers about their benefits, leading to a majority of mobile testing being performed manually. With the goal of helping researchers and practitioners - who design approaches supporting mobile testing - to understand developer's needs, we analyzed survey responses from 102 open source contributors to Android projects about their practices when performing testing. The survey focused on questions regarding practices and preferences of developers/testers in-the-wild for (i) designing and generating test cases, (ii) automated testing practices, and (iii) perceptions of quality metrics such as code coverage for determining test quality. Analyzing the information gleaned from this survey, we compile a body of knowledge to help guide researchers and professionals toward tailoring new automated testing approaches to the need of a diverse set of open source developers.Comment: 11 pages, accepted to the Proceedings of the 33rd IEEE International Conference on Software Maintenance and Evolution (ICSME'17

Enhancing Formal Modelling Tool Support with Increased Automation

Progress report for the qualification exam report for PhD Student Kenneth Lausdahl. Initial work on enhancing tool support for the formal method VDM and the concept of unifying a abstract syntax tree with the ability for isolated extensions is described. The tool support includes a connection to UML and a test automation principle based on traces written as a kind of regular expressions

System to Software Integrity: A Case Study

It is widely acknowledged that the main source of cost for developing high-integrity software systems is their verification. A significant portion of this verification cost is spent assessing that software complies with its requirements. Over the years several different methods have been developed to address this issue, in particular: testing, peer reviews, formal verification and automatic code generation. It is more and more frequent that these verification strategies are mixed within the same system, so as to adopt the most appropriate one for each component. This increases the complexity of the integration phase because it has to cope with multiple formalisms, development and verification methods. Our goal is to propose a pragmatic process to integrate components developed using different methods into a single system and demonstrate that properties already verified for each component in isolation are preserved in their composition. This process leverages AADL as a pivotal modeling language for system specification and relies on specific verifications between the latter and the components developed using heterogeneous modeling and programming languages, namely Simulink for computation intensive parts and Ada/SPARK 2014 for other components. Our paper proceeds as follows. First we provide a high-level overview of our approach and enumerate the current methods for addressing the property preservation problem. Then we illustrate practically our approach using the Nose Gear Challenge problem, a simplified yet complete example of a high-integrity real-time system. We then conclude by comparing our approach to the state of the art