Improvements of and Extensions to FSMWeb: Testing Mobile Apps

A mobile application is a software program that runs on mobile device. In 2017, 178.1 billion mobile apps downloaded and the number is expected to grow to 258.2 billion app downloads in 2022 [19]. The number of app downloads poses a challenge for mobile application testers to find the right approach to test apps. This dissertation extends the FSMWeb approach for testing web applications [50] to test mobile applications (FSMApp). During the process of analyzing FSMWeb how it could be extended to test Mobile Apps, a number of shortcomings were detected which we improved upon. We discuss these first. We present an approach to generate black-box tests to test fail-safe behavior for web applications. We apply the approach to a large commercial web application. The approach uses a functional (behavioral) model to generate tests. It then determines at which states in the execution of behavioral test failures can occur and what mitigation requirements need to be tested. Mitigation requirements are used to build mitigation models for each failure type. From those mitigation models failure mitigation tests are generated. Next, this dissertation provides an approach for selective black-box model-based fail-safe regression testing for web applications. It classifies existing tests and test requirements as reusable, retestable, and obsolete. Removing reusable test requirements reduces test requirements between 49% to 65% in the case study. The approach also uses partial regeneration for new tests wherever possible. Third, we present the new FSMApp approach to test mobile applications and compare the approach with several other approaches [88, 37]. A number of case studies explore applicability, scalability, effectiveness, and efficiency of FSMApp with other approaches. Future work makes suggestion on how to improve test generation and execution efficiency with FSMApp

Fail-Safe Testing of Web Applications

This dissertation introduces an approach to generate tests to test fail-safe behavior for web applications. We apply the approach to a commercial web application. We build models for both behavioral and mitigation requirements. We create mitigation tests from an existing functional black box test suite by determining failure type and points of failure in the test suite and weaving required mitigation based on weaving rules to generate a test suite that tests proper mitigation of failures. A genetic algorithm (GA) is used to determine points of failure and type of failure that needs to be tested. Mitigation test paths are woven into the behavioral test at the point of failure based on failure specific weaving rules. A simulator was developed to evaluate choice of parameters for the genetic algorithm. We showed how to tune the fitness function and performed tuning experiments for GA to determine what values to use for exploration weight and prospecting weight. We found that higher defect densities make prospecting and mining more successful, while lower mitigation defect densities need more exploration. We compare efficiency and effectiveness of the approach. First, the GA approach is compared to random selection. The results show that the GA performance was better than random selection and that the approach was robust when the search space increased. Second, we compare the GA against four coverage criteria. The results of comparison show that test requirements generated by a genetic algorithm (GA) are more efficient than three of the four coverage criteria for large search spaces. They are equally effective. For small search spaces, the genetic algorithm is less effective than three of the four coverage criteria. The fourth coverage criteria is too weak and unable to find all defects in almost all cases. We also present a large case study of a mortgage system at one of our industrial partners and show how we formalize the approach. We evaluate the use of a GA to create test requirements. The evaluation includes choice of initial population, multiplicity of runs and a discussion of the cost of evaluating fitness. Finally, we build a selective regression testing approach based on types of changes (add, delete, or modify) that could occur in the behavioral model, the fault model, the mitigation models, the weaving rules, and the state-event matrix. We provide a systematic method by showing the formalization steps for each type of change to the various models

A Unified Approach to Regression Testing for Mobile Apps

Mobile Applications have been widely used in recent years daily all over the world and are essential in our personal lives and at work. Because Mobile Applications update frequently, it is important that developers perform regression testing to ensure their quality. In addition, the Mobile Applications market has been growing rapidly, allowing anyone to write and publish an application without appropriate validation. A need for regression testing has arisen with the growth of different Mobile Apps and the added functionalities and complexities. In this dissertation, we adapted the FSMWeb [14] approach for selective regression testing to allow for selective regression testing of Mobile Apps. We applied rules to classify the original set of tests of the Mobile App into obsolete, retestable, and reusable tests based on the types of changes to the model of Mobile Apps. New tests are added to cover portions that have not been tested. As regression test suites change, we want to ensure that required tests are included to satisfy testing criteria, but also that redundant tests are removed, so as not to bloat the regression tests suite. In the dissertation, we developed a test case minimization approach for FSMApp, based on concept analysis that removes redundant test cases. Next, we proposed an approach to prioritize test cases for Mobile Apps. Naturally, it is desirable to select those test cases that are most likely to reveal defects in the App under test. We prioritized test paths for Mobile Apps based on input complexity, since more inputs might be associated with a more complex functionality which in turn would make it more fault-prone. As we knew, regression testing is an important activity in software maintenance and enhancement. Combining several regression testing techniques can lead to a more efficient and effective regression test suite. In this dissertation, we presented guidelines for combining regression testing approaches based on a systematic approach. We outlined all possible situations that can occur and showed how each of them influences which combination to use. Also, we validated the newly proposed regression testing approaches for Mobile Apps and the guidelines for combining regression testing approaches via a case study. The results show that FSMApp approaches are applicable, efficient, and effective

Model-Based Testing of Smart Home Systems Using EFSM, CEFSM, and FSMApp

Smart Home Systems (SHS) are some of the most popular Internet of Things (IoT) applications. In 2021, there were 52.22 million smart homes in the United States and they are expected to grow to 77.1 million in 2025 [71]. According to MediaPost [74], 69 percent of American households have at least one smart home device. The number of smart home systems poses a challenge for software testers to find the right approach to test these systems. This dissertation employs Extended Finite State Machines (EFSMs) [6, 24, 105], Communicating Extended Finite State Machines (EFSMs) [68] and FSMApp [10] to generate reusable test-ready models of smart home systems. We present an approach to create reusable test-ready models of smart home systems using EFSMs to model device components (Sensor, Controller and Actuator), EFSMs to model single devices in the SHS and the interaction between the devices. We adopted Al Haddad’s [10] FSMApp approach to model and test the mobile application that controls the SHS. These reusable test-ready models were used to generate tests. This dissertation also addresses evolution in smart home systems. Evolution is classified into three categories: adding a new device, updating an excising device or removing one. A method for selective black-box model-based regression testing for these changes was proposed

Fail-Safe Test Generation of Safety Critical Systems

This dissertation introduces a technique for testing proper failure mitigation in safety critical systems. Unlike other approaches which integrate behavioral and failure models, and then generate tests from the integrated model, we build safety mitigation tests from an existing behavioral test suite, using an explicit mitigation model for which we generate mitigation paths which are then woven at selected failure points into the original test suite to create failure-mitigation tests (safety mitigation test)

A Mapping Study of scientific merit of papers, which subject are web applications test techniques, considering their validity threats

Progress in software engineering requires (1) more empirical studies of quality, (2) increased focus on synthesizing evidence, (3) more theories to be built and tested, and (4) the validity of the experiment is directly related with the level of confidence in the process of experimental investigation. This paper presents the results of a qualitative and quantitative classification of the threats to the validity of software engineering experiments comprising a total of 92 articles published in the period 2001-2015, dealing with software testing of Web applications. Our results show that 29.4% of the analyzed articles do not mention any threats to validity, 44.2% do it briefly, and 14% do it judiciously; that leaves a question: these studies have scientific value

Testing Mobile Web Applications for W3C Best Practice Compliance

Adherence to best practices and standards when developing mobile web applications is important to achieving a quality outcome. As smartphones and tablet PCs continue to proliferate in the consumer electronics market, businesses and individuals are increasingly turning from the native application paradigm to HTML 5-based web applications as a means of software development and distribution. With an everincreasing reliance by users on the correct functioning of such applications, the requirement for stringent and comprehensive quality assurance measures is also brought sharply into focus. This research investigates the increasing trend towards mobile web application development in the mobile software domain, and assesses the requirement for an automated approach to best practice validation testing for mobile web applications. Contemporary approaches to automated web application testing are examined, with particular emphasis on issues relating to mobile web application tests. The individual guidelines proposed by the W3C Mobile Web Application Best Practices are analysed and where applicable automated conformance tests are implemented in a customised testing tool. A range of mobile web applications are tested using this tool in order to examine the extent to which implementation of the tested-for guidelines is detected. Automated tests were successfully implemented in respect of nearly 60% of the best practices