Design and implementation of a t-way test data generation strategy with automated execution tool support

To ensure an acceptable level of quality and reliability of a typical software product, it is desirable to test every possible combination of input data under various configurations. However, due to the combinatorial explosion problem, exhaustive testing is practically impossible. Resource constraints, cost factors, and strict time-to-market deadlines are some of the main factors that inhibit such a consideration. Earlier research has suggested that a sampling strategy (i.e., one that is based on a t-way parameter interaction) can be effective. As a result, many helpful t-way sampling strategies have been developed and can be found in the literature. Several advances have been achieved in the last 15 years, which have, in particular, served to facilitate the test planning process by systematically minimizing the test size required (based on certain t-way parameter interactions). Despite this significant progress, the integration and automation of strategies (from planning process to execution) are still lacking. Additionally, strategizing to sample (and construct) a minimum test set from the exhaustive test space is an NP-complete problem; that is, it is often unlikely that an efficient strategy exists that could regularly generate an optimal test set. Motivated by these challenges, this paper discusses the design, implementation, and validation of an efficient strategy for t-way testing, the GTWay strategy. The main contribution of GTWay is the integration of t-way test data generation with automated (concurrent) execution as part of its tool implementation. Unlike most previous methods, GTWay addresses the generation of test data for a high coverage strength (t > 6)

ChimpCheck: Property-Based Randomized Test Generation for Interactive Apps

We consider the problem of generating relevant execution traces to test rich interactive applications. Rich interactive applications, such as apps on mobile platforms, are complex stateful and often distributed systems where sufficiently exercising the app with user-interaction (UI) event sequences to expose defects is both hard and time-consuming. In particular, there is a fundamental tension between brute-force random UI exercising tools, which are fully-automated but offer low relevance, and UI test scripts, which are manual but offer high relevance. In this paper, we consider a middle way---enabling a seamless fusion of scripted and randomized UI testing. This fusion is prototyped in a testing tool called ChimpCheck for programming, generating, and executing property-based randomized test cases for Android apps. Our approach realizes this fusion by offering a high-level, embedded domain-specific language for defining custom generators of simulated user-interaction event sequences. What follows is a combinator library built on industrial strength frameworks for property-based testing (ScalaCheck) and Android testing (Android JUnit and Espresso) to implement property-based randomized testing for Android development. Driven by real, reported issues in open source Android apps, we show, through case studies, how ChimpCheck enables expressing effective testing patterns in a compact manner.Comment: 20 pages, 21 figures, Symposium on New ideas, New Paradigms, and Reflections on Programming and Software (Onward!2017

Semantics-based Automated Web Testing

We present TAO, a software testing tool performing automated test and oracle generation based on a semantic approach. TAO entangles grammar-based test generation with automated semantics evaluation using a denotational semantics framework. We show how TAO can be incorporated with the Selenium automation tool for automated web testing, and how TAO can be further extended to support automated delta debugging, where a failing web test script can be systematically reduced based on grammar-directed strategies. A real-life parking website is adopted throughout the paper to demonstrate the effectivity of our semantics-based web testing approach.Comment: In Proceedings WWV 2015, arXiv:1508.0338