GUI Testing Using Computer Vision

Testing a GUI's visual behavior typically requires human testers to interact with the GUI and to observe whether the expected results of interaction are presented. This paper presents a new approach to GUI testing using computer vision for testers to automate their tasks. Testers can write a visual test script that uses images to specify which GUI components to interact with and what visual feedback to be observed. Testers can also generate visual test scripts by demonstration. By recording both input events and screen images, it is possible to extract the images of components interacted with and the visual feedback seen by the demonstrator, and generate a visual test script automatically. We show that a variety of GUI behavior can be tested using this approach. Also, we show how this approach can facilitate good testing practices such as unit testing, regression testing, and test-driven development.National Science Foundation (U.S.). (Grant number IIS-0447800)Quanta Computer (Firm) (TParty project

Working Notes from the 1992 AAAI Workshop on Automating Software Design. Theme: Domain Specific Software Design

The goal of this workshop is to identify different architectural approaches to building domain-specific software design systems and to explore issues unique to domain-specific (vs. general-purpose) software design. Some general issues that cut across the particular software design domain include: (1) knowledge representation, acquisition, and maintenance; (2) specialized software design techniques; and (3) user interaction and user interface

Automated Testing For Software Process Automation

Robotic Process Automation is a way of automatizing business processes within short timespans. At the case company the initial automation step is implemented by business experts, rather than software developers. Using developers with limited software engineering experience allows for high speed but raises concerns of automation quality. One way to reduce these concerns is extensive testing, which takes up much time for integration developers. The aim of this thesis is to increase the quality of the development process, while minimizing impact on development time through test automation. The research is carried out as a part of the Robotic Process Automation project at the case company. The artifact produced by this thesis is a process for automatically testing software automation products. Automated testing of software automation solutions was found to be technically feasible, but difficult. Robotic process automation provides several novel challenges for test automation, but certain uses such as regression and integration testing are still possible. Benefits of the chosen approach are traceability for quality, developer confidence and potentially increased development speed. In addition, test automation facilitates the adoption of agile software development methods, such as continuous integration and deployment. The usage of continuous integration in relation to Robotic Process Automation was demonstrated via a newly developed workflow.Ohjelmistoautomaatio on nopea tapa automatisoida liiketoimintaprosessien rutiineja. Tapausyrityksessä automaation luovat ohjelmistonkehittäjien sijasta liiketoiminnan asiantuntijat. Käyttämällä alkukehittäjiä, joilla on vähäisesti kokemusta ohjelmistokehityksestä, saadaan nopeita ratkaisuja, mutta samalla yrityksellä on huolia laadusta. Laatua voidaan mitata testaamalla automaatioratkaisuja laajasti, mutta tähän menee huomattavasti aikaa. Tämän tutkielman tarkoituksena on testiautomaatiota käyttämällä nostaa kehitysprosessin laatua ilman että työmäärä kasvaa merkittävästi. Tutkimus suoritettiin osana tapausyrityksen ohjelmistorobotiikkaprojektia. Tutkielmassa luotiin prosessi, jossa automaattisesti testataan ohjelmistoautomaatioprosesseja. Testaus todettiin tutkimuksessa mahdolliseksi mutta käytännössä haasteelliseksi. Testauksessa ilmeni useita ongelmia, mutta muutamat ratkaisut kuten regressio- ja integraatiotestaus todettiin kuitenkin hyödyllisiksi. Lähestymistavan hyödyiksi todettiin laadun jäljitettävyyden, kehittäjien itseluottamuksen ja kehitysnopeuden kasvu. Lisäksi testiautomaatio mahdollistaa nykyaikaisten ketterien menetelmien kuten jatkuvan integraation käytön. Jatkuvan integraation käyttömahdollisuus demonstroitiin uudistetulla työtavalla

Evolving Legacy System\u27s Features into Fine-grained Components Using Regression Test-Cases

Because many software systems used for business today are considered legacy systems, the need for software evolution techniques has never been greater. We propose a novel evolution methodology for legacy systems that integrates the concepts of features, regression testing, and Component-Based Software Engineering (CBSE). Regression test suites are untapped resources that contain important information about the features of a software system. By exercising each feature with its associated test cases using code profilers and similar tools, code can be located and refactored to create components. The unique combination of Feature Engineering and CBSE makes it possible for a legacy system to be modernized quickly and affordably. We develop a new framework to evolve legacy software that maps the features to software components refactored from their feature implementation. In this dissertation, we make the following contributions: First, a new methodology to evolve legacy code is developed that improves the maintainability of evolved legacy systems. Second, the technique describes a clear understanding between features and functionality, and relationships among features using our feature model. Third, the methodology provides guidelines to construct feature-based reusable components using our fine-grained component model. Fourth, we bridge the complexity gap by identifying feature-based test cases and developing feature-based reusable components. We show how to reuse existing tools to aid the evolution of legacy systems rather than re-writing special purpose tools for program slicing and requirement management. We have validated our approach on the evolution of a real-world legacy system. By applying this methodology, American Financial Systems, Inc. (AFS), has successfully restructured its enterprise legacy system and reduced the costs of future maintenance

Assessing the Reuse Potential of Objects.

In this research, we investigate whether reusable classes can be characterized by object-oriented (OO) software metrics. Three class-level reuse measures for the OO paradigm are defined: inheritance-based reuse, inter-application reuse by extension, and inter-application reuse as a server. Using data from a software company, we collected metrics on Smalltalk classes. Among the 20 metrics collected are cyclomatic complexity, Lorenz complexity, lines of code, class coupling, reuse ratio, specialization ratio and number of direct subclasses. We used stepwise regression to derive prediction models incorporating the 20 metrics as the independent variables and the reuse measures, applied separately, as the dependent variable. Inheritance-based reuse and inter-application reuse by extension can be predicted using a subset of the 20 metrics. Two prediction models for inheritance-based reuse and inter-application reuse by extension were validated using a new set of 310 Smalltalk and VisualAge applications and subapplications. Validation results show that it is possible to predict whether a class from one application can be reused by extension in another application. We also conducted a t-test to test whether the mean metric values between reusable and non-reusable classes are the same. Results suggest that there exists significant differences in the mean metric values between the reusable and non-reusable classes

Maintenance of Automated Test Suites in Industry: An Empirical study on Visual GUI Testing

Context: Verification and validation (V&V) activities make up 20 to 50 percent of the total development costs of a software system in practice. Test automation is proposed to lower these V&V costs but available research only provides limited empirical data from industrial practice about the maintenance costs of automated tests and what factors affect these costs. In particular, these costs and factors are unknown for automated GUI-based testing. Objective: This paper addresses this lack of knowledge through analysis of the costs and factors associated with the maintenance of automated GUI-based tests in industrial practice. Method: An empirical study at two companies, Siemens and Saab, is reported where interviews about, and empirical work with, Visual GUI Testing is performed to acquire data about the technique's maintenance costs and feasibility. Results: 13 factors are observed that affect maintenance, e.g. tester knowledge/experience and test case complexity. Further, statistical analysis shows that developing new test scripts is costlier than maintenance but also that frequent maintenance is less costly than infrequent, big bang maintenance. In addition a cost model, based on previous work, is presented that estimates the time to positive return on investment (ROI) of test automation compared to manual testing. Conclusions: It is concluded that test automation can lower overall software development costs of a project whilst also having positive effects on software quality. However, maintenance costs can still be considerable and the less time a company currently spends on manual testing, the more time is required before positive, economic, ROI is reached after automation

Library of model components for process simulation relevant to production activities, Prototype 1 versions

Production Economics,

Techniques for Efficient and Effective Mobile Testing

The booming mobile app market attracts a large number of developers. As a result, the competition is extremely tough. This fierce competition leads to high standards required for mobile apps, which mandates efficient and effective testing. Efficient testing requires little effort to use, while effective testing checks that the app under test behaves as expected. Manual testing is highly effective, but it is costly. Automatic testing should come to the rescue, but current automatic methods are either ineffective or inefficient. Methods using implicit specifications – for instance, “an app should not crash” for catching fail-stop errors – are ineffective because they cannot find semantic problems. Methods using explicit specifications such as test scripts are inefficient because they require huge developer effort to create and maintain specifications. In this thesis, we present our two approaches for solving these challenges. We first built the AppDoctor system which efficiently tests mobile apps. It quickly explores an app then slowly but accurately verifies the potential problems to identify bugs without introducing false positives. It uses dependencies discovered between actions to simplify its reports. Our second approach, implemented in the AppFlow system, leverages the ample opportunity of reusing test cases between apps to gain efficiency without losing effectiveness. It allows common UI elements to be used in test scripts then recognizes these UI elements in real apps using a machine learning approach. The system also allows tests to be specified in reusable pieces, and provides a system to synthesize complete test cases from these reusable pieces. It enables robust tests to be created and reused across apps in the same category. The combination of these two approaches enables a developer to quickly test an app on a great number of combinations of actions for fail-stop problems, and effortlessly and efficiently test the app on most common scenarios for semantic problems. This combination covers most of her test requirements and greatly reduces her burden in testing the app